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About this document 


This report includes a summary of findings and an appraisal of existing models used across 
the world to involve the public in heath technology assessment. 

It is intended to inform discussions of the Medicare Benefits Schedule Review Taskforce 
about the 'improvement of our Medicare Benefits Schedule (MBS)' in order to ensure that it is 
'consistent with the latest clinical practice, or the best value healthcare' 1 . This report focusses 
on ways of improving the way the public are involved in the process of reviewing MBS Items. 

This report also contains an Appendix which includes the search report from the literature 
review, a summary of the data extracted, links to the full data set extracted from the rapid 
review and additional information 

The report contains a level of detail intended to reflect the complex nature of the subject. As 
an aid, the report includes a one page summary of the entire report, and an executive 
summary of key learning points that have been extracted from the report. 

Referencing 

Documents from the literature review will be referenced throughout this report by giving the 
author or organisation's name, the unique reference number (created during this literature 
review) and a footnote URL to the original document. For example, Nunn, (AOOO ) 2 Other 
documents and citations will be cited with a footnote. Where possible URLs will direct to a 
publicly accessible document. Where a document is not in the public domain (for example, a 
journal article behind a 'paywall'), a link will be given to this document in an online shared 
folder created for this report. 

Please note, copyrighted material is hosted for education and research purposes and 
appropriate arrangements should be made if disseminating in the public domain. 

Archived links 

Where possible, resources in the public domain reference pages on 'The Internet Archive' 3 , a 
not-for-profit organisation which provides a free service which stores web-pages and 
documents as they were at certain points in time. While ensuring that references are future- 
proofed, the link also contains information about when the resource was accessed and 
'saved', and the original full URL. For example, the date saved is highlighted here in red, the 
original hyperlink in purple: 


1 http://web.archive.org/web/201 51 1 170441 41 /http://www.health. aov.au/internet/main/publishing.nsf 
/Content/922CB2933B0F1 645CA257EC1 001 D5C1 2/$File/MBS%20Review Consultation%20paper Ove 
rview_FINAL.pdf 

2 [example hyperlink] 

3 https://archive.org/web/ 


http://web.archive.org/web/201 51 120043049/http://www. health. aov.au/internet/main/publishina.nsf/ 
Content/M BSReviewTaskforce 


Audience 

Although this document has a specific audience, where possible it has been written in 'lay' 
language so that members of the public and others with no specialist knowledge may be able 
to understand it. 

One page summary 


Section title: Exploring definitions and terms 

This section explores the variety of words and terms used throughout the English speaking 
world to describe 'consumer engagement' and associated concepts. 

Key learning point 

Involving the public in creating and agreeing the language used to describe engagement, 
participation and involvement will ensure that it is accessible, inclusive and therefore effective. 

Section title: Detailed appraisal of five relevant 
models 

This sections examines five internationally relevant models in detail, appraising the design 
and, where appropriate, key elements and limitations. 

Key learning point 

While these models represent a number of different ways of involving the public and 
evaluating involvement, there is no evidence-based validation for any of the models and 
therefore no endorsed option. 

Section title: Relevant generic models of 
involvement 

This section contains a summary of models of public involvement and engagement which, 
while relevant to health technology assessment, are more generic. 

Key learning point 

While there are a number of relevant international models, there is significant variation in the 
definitions of certain words used, such as engagement, and the purpose of any involvement or 
engagement. The Health Research Authority (UK) provides the best example of an 
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organisation clearly articulating what they mean by public involvement, thus providing a 
robust linguistic framework of reference for future involvement models 4 

Section title: Significant themes 

The significant themes identified were 'social and ethical issues', 'public involvement', 
'learning, training, education and development'. 

Key learning point 

While there may be challenges to including and involving everyone in the appraisal process, 
'tensions between equity and efficiency' 5 , there are ways of overcoming some of these 
challenges. For example, creating inclusive learning and development opportunities for the 
public. 


4 http://web.archive.org/web/201 51 1 20041 757/http://www.hra.nhs.uk/documents/201 3/1 0/hra- 
public-involvement-strateqy-circulation-september-201 3.pdf (page 1 1 ) 

5 http://web.archive.org/web/201 51 1 2401 2937/http://www. nice. orq.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-eguity-efficiencv.pdf 
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Executive Summary 


Introduction 

This report summarises a review of international literature about ways of involving the public 
in heath technology assessment. It includes a summary of over fifty documents including 
journal articles, reports and webpages. It has also been informed by interviews with subject 
area experts. 

In order to frame the literature, the first section 'Exploring definitions and terms' considers 
variations in the way definitions and terms are used to describe public involvement across the 
English speaking world. This section concludes with a diagram called 'Mapping the definitions' 
which attempts to visualise all the terms used. 

In the next section, 'Detailed appraisal of five relevant models', relevant models from across the 
world have been summarised and appraised with key findings and limitations of each relevant 
model examined. 

In order to provide context for the models, the 'Significant themes' section identifies important 
themes which emerged from the literature review. Each theme is explored in detail, identifying 
significant guotations and key learning points in order to help appraise and evaluate the 
models further. This section also contains an examination of the 'Social determinants of 
involvement', including diagrams which summarise the findings from the review. These 
diagrams may also support the MBS Review Taskforce in appraising any plans for public 
involvement. 

Below is a summary of each section divided into a 'Summary' and 'Key learning points' 

Summary of 'Exploring definitions and terms' 

This section explores the variety of words and terms used throughout the English speaking 
world to describe 'consumer engagement' and associated concepts such as 'public 
involvement'. 

This section begins with an examination of language used in existing MBS resources in the 
public domain, focussing in particular on the uses of the word 'stakeholder 1 . 

The section then examines the differences between associated terms such as 'engagement', 
'participation' and 'involvement'. It considers the guestion who is 'involved' or 'engaged'? The 
section concludes with a consideration of international literature which suggests that the term 
'public' is most helpful when involving patients, consumers, carers and other people who use 
or may use of services in the future. 

Key learning point 
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Involving the public in the process of creating and agreeing the language which will be used to 
describe engagement, participation and involvement will ensure that it is accessible, inclusive 
and therefore effective. 



Summary of 'Detailed appraisal of five relevant 
models' 


This sections examines five internationally relevant models in detail, appraising the design 
and, where appropriate, key elements and limitations. Below are short summaries of each 
model. 

Key learning point of 'Detailed appraisal of five relevant models' 

While these models present a number of different ways of involving the public and evaluating 
involvement there is no evidence-based validation of models for public involvement. 

Summary of ' Role of patient and public participation in Health Technology 
Assessment and coverage decisions' ( Menon , A037 6 ) 

• Menon highlights the need for gualitative methodologies to compliment guantitative 
ones and identifies key stages of the HTA process, explaining the importance of public 
involvement at each stage 

• Menon discusses the importance of public involvement not only to the public, but also 
to uphold the validity of the HTA process (and thus funding) itself 

• Menon offers a framework for analysing public involvement in HTA. This framework 
'recognizes the various facets of possible public engagement in HTA as three 
domains': 

o the policy making domain 
o the organizational domain 
o the research domain. 


Key Issues iden tified by Menon 

• Patients and members of the public may have different roles, and should be 
considered separately when they are to be involved in health technology assessment. 

• There are methodological challenges with respect to how information on the 'patient 
experience' should be collected and synthesized. 

• Since many patient organizations receive funding from industry sources there is 
concern about the objectivity of the information these organizations provide. 

• Since not all patients share the same views it is important to canvas multiple patients 
and patient organizations. 

• Despite a lack of evaluative evidence on the comparative merit of different approaches 
to engaging the public and patients there seems to be a general consensus that 
obtaining their perspectives is important. 


6 https://drive.google.com/open7icHOBwQTUgazNAy5UEROckEwbl96VEE 
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Summary of A Pilot Study to Identify Areas for Further Improvements in 
Patient and Public Involvement In Health Technology Assessments for 
Medicines' (Messina, A022 f 

Key elements 

The pilot study identified three key areas for 'further advancement' in the field of 'public input' 
in the HTA process: 

Advancement 1 : industry could help bring the patient perspective into the HTA process 
through incorporating patient experiences early in the drug development process and by 
including gualitative research on patient experiences in HTA dossiers. In addition 'It would be 
helpful to the HTA process if patients' needs and views were considered in the design of 
phase III research trials conducted by industry'. 

Advancement 2: recognizing and supporting the role of patient advocacy groups, and making 
use of their access to the genuine patient perspective and experience of living with the 
condition in guestion. 

Advancement 3: the continuous development of HTA systems and processes to better 
facilitate involvement, increase transparency and feedback, explore new options for reaching 
patients, and focus on creating an active and informed health consumer. 

The paper includes the following 'key points for decision makers': 

• The health technology assessment (HTA) of medicines should incorporate the 
patient's experience of what it is like living with the disease and views on the 
technology from the patient's perspective 

• Industry can assist in acguiring patient voices by including patient views early on in 
drug development 

• Increasing transparency and feedback, and exploring new options for reaching 
patients are needed 

The paper also notes that 'the exact point to seek input from patients and the public is 
controversial'; however, it suggests that 'those involved within HTAs agree that patients/the 
public are usually involved too late in the process'. 


7 https://drive.aooqle.com/open7icHOBwQTUqazNAv5WEZNN2lveilOQkU 


Summary of 'Involving patients in the early stages of health technology 
assessment (HTA): a study protocol 1 ( Gagnon , A015) 8 

Key elements 

• Recommends that a third of the participants involved in the deliberation process of 
prioritizations topics for HTA must be patients. 

• Advocates patient involvement in all stages of the HTA process as patient involvement in 
early stages make assessments 'more relevant and acceptable' (Gagnon, A01 5 9 ). 

• Claims that methods of involvement 'that involve face-to face interaction between 
participants from the public and decision makers also generate greater participant 
satisfaction, both in terms of process and results' (Gagnon, A01 5 10 ). 

• Based on learning from the UK the authors suggest patients should 'receive training in HTA 
and evidence-informed healthcare before the prioritisation meetings'(Gagnon, A01 5 11 ). 

• Patients are involved in the consultation and reviewing process of a potential HTA topic as 
well as development of the assessment plan. 


8 https://drive.aooale.eom/file/d/QBwQTUaazNAv5b3huQ2JWS1 pwT m8 

9 https://drive.aooale.eom/file/d/0BwQTUaazNAv5b3huQ2JWS1 pwT m8 

10 https://drive.aooale.eom/file/d/0BwQTUaazNAv5b3huQ2JWS1 pwTm8 

11 https://drive.aooale.eom/file/d/0BwQTUaazNAv5b3huQ2JWS1 pwTm8 
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Summary of 'Public Participation Methods : A Framework for Evaluation' 
(Rowe, A051) 12 

Key elements 

Rowe proposes a framework for evaluating the 'effectiveness' of the various models of 
participation. 

The evaluation criteria are divided into: 

• Acceptance criteria - related to the effective construction and implementation of a 
procedure 

• Process criteria - related to the potential public acceptance of a procedure 

Rowe provides a detailed commentary about each method and summarises the findings in 
the following table: 



Referenda 

Public 

Hearings 

Public 

Opinion 

Survey 

Negotiated 

Rule 

Making 

Consensus 

Conference 

Citizens’ 

Jury/Panel 

Citizen 

Advisory 

Committee 

Focus 

Groups 

Acceptance criteria 

Representativeness 
of participants 

High (assum- 
ing full turn- 
out at poll) 

Low 

Generally 

high 

Low 

Moderate Moderate 

(limited by (limited by 

small sample) small sample) 

Moderate 1 
to low 

1 

Moderate 
(limited by 
small sample) 

Independence of 
true participants 

High 

Generally 

low 

High 

Moderate 

High 

High 

Moderate 
(often relation 
to sponsor) 

High 

Early involvement? 

Variable 

Variable 

Potentially 

high 

Variable 

Potentially 

high 

Potentially 

high 

Variable but 
may be high 

Potentially 

high 

Influence on final 
policy 

High 

Moderate 

Indirect and 
difficult to 
determine 

High 

Variable 
but not 
guaranteed 

Variable 
but not 
guaranteed 

Variable 
but not 
guaranteed 

Liable to be 
indirect 

Transparency of 

process to the public 

High 

Moderate 

Moderate 

Low 

High 

Moderate 

Variable but 

Low 


often low 


12 


http://web.archive.org/web/201 51 1 1 8033233/http://www.en. ipea.aov.br/participacao/images/pdfs/pa 
rticipacao/2000 0 /o20public 0 /o20participation%20methods.pdf 



Referenda 

Public 

Hearings 

Public 

Opinion 

Survey 

Negotiated 

Rule 

Making 

Consensus 

Conference 

Citizens' 

Jury/Panel 

Citizen 

Advisory 

Committee 

Focus 

Groups 

Process criteria 









Resource accessibility 

Low 

Low- 

moderate 

Low 

High 

High 

High 

Variable 

Low 

Task definition 

High 

Generally 

high 

Low 

High 

Generally 

high 

Generally 

high 

Variable but 
may be high 

Variable but 
may be high 

Structured decision 
making 

Low 

Low 

Low 

Moderate 

Moderate 
(influence of 
facilitator) 

Potentially 

high 

Variable 
(influence of 
facilitator) 

Low 

Cost-effectiveness 

Variable/low 

Low 

Potentially 

high 

Potentially 

high 

Moderate 
to high 

Moderate 
to high 

Variable 

Potentially 

high 
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Summary of the 'National Institute for Health and Care Excellence (NICE) 
process' (England and Wales) 

Key elements 

• Appraisal committee meetings are open to the public, which promotes public 
understanding of how evidence is assessed. 

• The Public Involvement Programme (PIP) assigns a Public Involvement Adviser to each 
appraisal and supports patient and carer consultee organisations, their representatives, 
and individual patients or carers throughout the appraisal. 

• When STA are taking place, Appraisal Consultation Documents (ACD) are published on the 
NICE website to allow access to the public (including experts from different fields) to 
submit their comments. A lay version of the ACD for the public (known as 'Information for 
the public 1 ) is published on its website. This inclusive process allows for different 
perspectives to be denoted and published for the Advisory Committee to take into 
account. 

• The public can appeal recommendations. 

• The Citizens Council 'is one of the few examples of broader public engagement (as 
opposed to specific patients or patient advocates)' (Messina, A022)13. 


13 https://drive.qooale.com/open7icNOBwQTUqazNAv5WEZNN2lveilOQkU 


Summary of 'Relevant generic models of involvement 


This section contains a summary of models of public involvement and engagement which, 
while relevant to health technology assessment, are more generic. 

The section 'Understanding whose voice needs to be heard' examines The ladder of 
participation', which can be a useful way of locating the degree of control the public have over 
a process, for example, the MBS Review. 

It then examines a number of terms such as 'expert 1 , 'advisor 1 , 'advocate' and 'partner' used in 
other models to help articulate different roles the public can have. 

The section 'National and international frameworks' examines frameworks for 'Patient Centred 
Care', 'public involvement', 'citizen involvement' and 'consumer engagement'. 

It includes an analysis of the International Association for Public Participation (IAP2) public 
participation spectrum 14 , the 'Engagement Cycle' 15 and the Health Issues Centre model for 
consumer engagement 16 . 

Key learning point 

While there are a number of relevant international models, there is significant variation in the 
definitions of certain words used, such as engagement, and the implicit purpose of any 
involvement or engagement. The Health Research Authority (UK) provides the best example of 
an organisation clearly articulating what they mean by public involvement, thus providing a 
robust linguistic framework of reference for future involvement models 17 . 

Summary of 'Significant themes' 

The significant themes identified were 'social and ethical issues', 'public involvement', 

'learning, training, education and development'. 

The section 'Social and ethical issues’ concludes that public involvement 'increases 
transparency' (Wortley, A002) 18 and the 'bringing together' of the public (or representatives) 
with subject experts in a transparent way is key to maintaining trust in the appraisal process. 


14 http://web.archive.ora/save/https://www. iap2.ora.au/resources/iap2s-public-participation-spectrum 

1 5 https://web.archive.org/web/201 501 14024258/http://enaaaementcvcle.ora/enaaaement-strateav- 
culture-and-svstems/ 

16 

http://web.archive.org/web/201 51 126042923/http://www. healthissuescentre.ora.au/imaaes/uploads/ 
resources/Consumer-participation-resource-for-researchers.pdf 

17 http://web.archive.org/web/201 51 120041 757/http://www.hra.nhs.uk/documents/201 3/1 0/hra- 
public-involvement-strateav-circulation-septennber-201 3.pdf /page 1 1 ) 

1 8 http://web.archive.org/web/201 51 1 06063004/http://www. biomed centrafcom/content/pdf/sl 2911- 
01 5-01 76-0. pdf 
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A report by the NICE Citizens Council states that the transparency of such processes can 
affect 'perceptions that values such as justice, equality and trust are not being upheld' 19 . 

The section 'Public involvement' examines the failure of evidence-based medicine to involve 
the public and considers the social determinants of involvement. 

This section concludes with two resources which have been adapted to support members of 
the Medicare Benefits Schedule Review Taskforce appraise their plans for involvement 

The final section is a 'A checklist of questions to ask at the start of public involvement in HTA' 

This short list of questions has been collated from knowledge gained from the literature 
review and is designed to stimulate the creation of a robust and transparent model of public 
involvement in the HTA process. 

Why do the public need to be involved? 

• Are there any exclusions, are there certain people you don't want to involve? Why? 

• Are there any people who might need additional support to be involved? What support 
might they need? 

What stage of your process are you planning to involve the public? 

• Are you planning to involve them from the start? If not, why? 

• Will you involve the public in helping you plan how you will involve the public? 

• Do you want different people involved at different stages? Why? 

How will you involve the public? 

• Have you involved the public in planning how you will involve the public? For example, 
have you asked members of the public if they think your plans may unintentionally exclude 
some people? 

• Have you got adequate resources or is there a limit to your involvement capacity? Have 
you been transparent about this limit? 

How will you let people know what you are doing? 

• Will the public be involved in helping you plan how you will let people know how to be 
involved (for example, recruitment or dissemination)? 

How will you evaluate your public involvement? 

• Have you planned adequate resources to evaluate involvement, including a longer term 
impact assessment? 

• Will you publish or share the results of your evaluation so others may share in any 
learning? 

Key learning point 


1 9 http://web.archive.org/web/201 51 1 20091 727/http://www. nice.org. uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-eguitv-efficiencv.pdf 


While there may be challenges to including and involving everyone in the appraisal process, or 
'tensions between eguity and efficiency ' 20 there are ways of overcoming some of these 
challenges. For example, creating inclusive learning and development opportunities for the 
public. 

Summary of appendices 

The appendices includes the following information about this literature search and report, 
including: 

• The method - a report of what terms were searched and where 

• A short discussion of inclusion and exclusion criteria 

• An appraisal of the literature review, including limitations 

• Output - a summary of the data extracted from the literature search 

It also contains a collation of anonymised interviews conducted with staff who work to involve 
the public in the work of NICE. 
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This is the end of the executive summary. 


20 http://web.archive.org/web/2Q1 51 1 2401 2937/http://www. nice. ora.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-equitv-efficiencv.pdf 
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Exploring definitions and terms 


Summary of language in Medicare Benefit Schedule - 
resources for the public 

This section examines the language used in existing MBS resources in the public domain. 

This slide below is taken from the 'MBS Review Stakeholder forum pack' and summarises 
ways of 'consulting stakeholders '. 21 

What are the most effective methods for consulting stakeholders? 

Consumer groups / focus groups 
Citizen juries 
Social media 

Engage through clinicians 
Case studies 
Issues / discussion areas 
— Expert consumer vs “normal” consumer 

- Co-design 

- Survey/email 

- Inform them beforehand 
— Potentially chair Discipline Groups 
Peak bodies / Colleges / Boards 
Case studies 
Scientific meetings 

Written submissions / consultations (2-3 month window) 

Issues / discussion areas 

- “Scare campaigns” 

— Nominate champions 
— Next generation of leaders 
Media - educate, engage 
Q&A response sheets 
Lots of various stakeholders listed! 



Consumer 


Clinicians 


Other 


21 

http://web.archive.org/web/201 51 1 1 7074506/http://www.health. qov.au/internet/main/publishing.nsf/ 
Content/95453EB597B6A358CA257E590024737B/$File/MBS%20Review 0 /o20Stakeholder°/o20Forum%2 
0Participant%20Pack o /o20-%20Perth%2025%20Julv%20201 5.pdf 



In a presentation for the MBS Review Taskforce, 'stakeholders are invited to actively 
contribute to the refinement of rules' 22 . 

The same presentation asks 'What are the most effective methods for consulting 
stakeholders?' and 'How should consumers be engaged in the review process?', presenting 
two parallel sets of language around 'consulting stakeholders' and 'engaging consumers'. 

The presentation also outlines how 'continuous dialogue' is taking place, with 'stakeholders' 
involved online (via a 'The Consultation Hub') and at 'stakeholder forums': 

Our continuous dialogue with stakeholders is happening via six channels 

Implications for stakeholders 


H Consultation Papers 
contain major 
questions and updates 

The Consultation Hub 
provides immediate 
opportunity for input 



■ The ongoing consultation paper is the best resource to learn about 
open questions and the Review’s approach 

■ Emerging recommendations will also be communicated for 
additional consultation prior to adoption 

■ All stakeholders are invited to contribute input into the consultation 
process via this online tool 

■ Feedback is channeled to the appropriate Committee 



Professional organi- 
sations are being 
continually engaged 


■ Clinical Committees and the Review Team are holding ongoing 
dialogues with medical Colleges, professional organisations and 
colleagues 


£ A v Stakeholder forums 
I 'Ur I seek live feedback 


■ Forums help to shape the direction of the Review 

■ Feedback will be integrated with online comments and shared with 
relevant Clinical Committees 



Distribution list 
members are kept up- 
to-date regularly 

MBS Review website 
provides key materials 


■ The Review team shares regular newsletters and updates via a 
central distribution list (sign up via website) 


■ The MBS Review website contains all relevant information about 
the program, including outcomes of these Forums 


I o 


Summary of literature review definitions 

This section summarises the different definitions of words found in the literature review. Each 
definition concludes with the meaning it will have in this document. 

Engagement 

"Generally public engagement is used as a broad term to cover a number of different types of 
dialogue; from basic information provision to more complex deliberation and collaboration. 


22 httDs://drive.aooale.com/open?id=QB7-zEXxcoXA4YTIkR0VQX1 FDYTF2b2swUnlwR0ZiZERVcia4 
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Many frameworks have been published to conceptualize the differences between types of 
engagement with most conceptualizing a continuum whereby each successive type 
represents an increasing degree of influence, commitment and/or participation” (Wortley, 
A002) 23 . 

Use in this document 

Engagement will be used to mean communicating with and informing people. Engaging 
people may lead to them knowing about and using services, or lead to them being 
involved in shaping them. 


Health Technology Assessment (HTA) 

The Australian Government defines Health Technology Assessment (HTA) as following: 

"HTA is a range of processes and mechanisms that use scientific evidence to assess the 
quality, safety, efficacy, effectiveness and cost effectiveness of health services. HTA is 
commonly applied to pharmaceuticals (including vaccines), diagnostic tests, medical devices, 
surgically implanted prostheses, medical procedures and public health interventions. The key 
questions that HTA typically aims to answer for each new health technology, in comparison to 
alternative interventions, are: 

• Is it safe? 

• Does it improve health outcomes? 

• Is it cost effective? 

Effective assessment of health technologies includes: 

• evaluating the comparative harms and benefits, using clinical evidence of patient safety, 
efficacy and clinical effectiveness; and 

• understanding the cause, origin and prevalence of disease and knowledge of best practice 
treatment pathways. 

A well-performing HTA system will: 

• facilitate patient access to cost-effective health technologies that improve health 
outcomes; 

• minimise the use of technologies that are ineffective or harmful; 

• contribute to value for money investments in health technology in the context of limited 
health care resources; 

• keep pace with evolving technologies, clinical practices and HTA methodologies; 

• provide clear information on processes, rules and outcomes to stakeholders; and 

• ensure the system is designed to achieve these outcomes in the most timely, effective, 
efficient and targeted way. 

HTA provides an assessment and prioritising of new technologies against existing health care 
interventions and other government funding priorities. This process results in the best value 
for money for the Australian community by considering both clinical effectiveness and cost 


23 http://web.archive.org/web/201 51 1 06063004/http://www. biomedcentral.com/content/pdf/sl 291 1 - 
01 5-01 76-0. pdf 


effectiveness. This ensures that taxpayers' money supports health care that achieves the 
maximum health improvement at the lowest cost. 

HTAalso provides health professionals, hospitals and consumers with information about 
safe, high-quality and clinically effective health technologies.” 24 

The European Network for HTA defines HTA as "...a multidisciplinary process that 
summarizes information about the medical, social, economic and ethical issues related to a 
health technology in a systematic, transparent, unbiased, robust manner.” (Menon, A037) 25 

Use in this document: HTA will refer to health technology assessement processes in 
general, with the MBS Review being specifically mentioned where relevant. 

Medicare Benefits Schedule (MBS) 

The MBS is a key component of the Medicare system. It lists all the services for which you can 
receive a rebate from Medicare. This includes 'out-of-hospital services provided by GPs, 
specialists, optometrists, and in some cases, dentists and other allied health practitioners, and 
where you are a private patient having in-hospital treatment (for example, if you have a baby 
as a private patient in a p private hospital)'. 26 

Use in this document 

'MBS' will be used to describe the Medicare Benefits Schedule. 

Who is 'involved' or 'engaged' - 'stakeholders', 
'patients' or 'public'? 

Introduction 

The terms 'involvement', 'engagement' and 'participation' have different meanings across the 
world and are sometimes used interchangeably. Similarly, 'stakeholders', 'patient', 'public' and 
'citizen' are used by many different organisations often conveying different meanings. 

This section is an attempt to briefly compare and summarise a few different uses of these 
terms. The next section attempts to visually map them in order to help articulate the 
sometimes nuanced differences. 

Comparison of terms 


24 

http://web.archive.org/web/201 51 1 06062545/http://www. health, gov.au/internet/hta/publishi 
ng.nsf/Content/about-1 

25 https://drive. google. com/open?id=OBwQTUgazNAy5UEROckEwbl96VEE 

26 

http://web.archive.org/web/201 51 1 170441 41 /http://www.health. gov.au/internet/main/publis 
hing.nsf/Content/922CB2933B0F1 645CA257EC1 001 D5C1 2/$File/MBS%20Review Consultati 
on%20paper_Qverview_FINAL.pdf 
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A distinction between 'stakeholder engagement' and 'community engagement' is made in 
published meeting notes of a Medicare Benefits Schedule Review Taskforce Meeting, which 
also distinguishes between 'craft groups' and 'the wider community' 27 . 

In a presentation to the MBS Review Taskforce, the public was implied in the phrase 'other 
affected stakeholders' 28 . 

The following quotations indicate the plurality in the meaning of the term stakeholder: 

• In HTA these stakeholders include health professionals, pharmaceutical companies, 
device manufacturers as well as the public and patient organizations.' (Wortley, A002) 29 . 

• A distinction is made between the public and patients - "ordinary or lay citizens rather than 
stakeholder groups such as patients or decision-makers.” (Wortley, A002) 30 

• The National Institute for Health and Care Excellence (NICE) distinguishes between 
'stakeholders' and 'individuals' 31 

NICE also defines a number of other related terms in an online glossary 32 . NICE uses the term 
'Patient expert' and defines it as someone who has: 

'used the technology either personally or as part of a representative group. Patient 
experts attend as individuals; they may be either somebody with personal experience 
of the condition, and if possible the technology, or a member of a patient and carer 
organisation for the condition being appraised' 33 

NICE also distinguishes between 'patients' and 'lay' people, with 'lay members' as part of an 
advisory committee: 

A lay member is a committee member with a patient, service user, carer or community 
background. The lay member's role is the same as other committee members, and 
additionally includes contributing a lay perspective and highlighting patient and carer 
issues. 


2Z 

http://web.archive.org/web/201 51 1 1 7090839/http://www. health, aov.au/internet/main/publishing.nsf/ 
Content/95453EB597B6A358CA257E590024737B/$File/MBSRT%20Meetina%20- 
%203%20Julv%20201 5%20-%200utcomes.pdf 

28 https://drive.aooale.com/open7icH0B7-zEXxcoXA4YTlkR0V0X1 FDYTF2b2swUnlwR0ZiZERVcia4 

29 http://web.archive.org/web/201 51 1 06063004/http://www. biomed central.com/content/pdf/sl 291 1 - 
01 5-0176-0.pdf 

30 http://web.archive.org/web/201 51 1 06063004/http://www. biomed central.com/content/pdf/sl 291 1 - 
015-0176-0.pdf 

31 

http://web.archive.org/web/201 51 1 20082306/https://www.nice. org.uk/article/pma1 9/chapter/qlossar 

y 

32 

http://web.archive.org/web/201 51 1 20082306/https://www.nice. org.uk/article/pma1 9/chapter/qlossar 

y 

33 

http://web.archive.org/web/201 51 1 20082306/https://www. nice.orq.uk/article/pmg1 9/chapter/qlossar 
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NICE also distinguishes between 'patients, carers and the public', stating they are 'committed 
to involving patients, carers and the public in the development of its guidance and other 
products ' 34 

NICE defines a 'carer' as 'a person who provides unpaid care by looking after a relative, friend 
or partner who needs support because of ill health, frailty or disability'. 

The Canadian Agency for Drugs and Technologies in Health (CADTH) distinguishes between 
'patients', 'stakeholders' and 'anyone with an interest' in their work 35 : 

'Listening to patients is a vital part of CADTH's work. We recognize that patients have 
unigue knowledge about what it is like to live with a specific disease or medical 
condition. They can describe advantages and disadvantages of current therapies, 
which may not be reported in published literature. And they can tell us what they would 
most value from a new treatment. We also know that there may be other stakeholders 
— individuals and organizations — with important insights to contribute to our work... 
Eligibility to provide feedback varies depending on the CADTH product, but it is often 
open to anyone with an interest in the work.' 

The Scottish Medicines Consortium (SMC) has a webpage titled 'public involvement', but 
doesn't use the term public on the page, with the heading 'Capturing the patient and carer 
voice 36 ': 


Understanding the experiences of patients, their families and carers is a key element in 
our decision making process. It is important for SMC members to fully understand 
how a new medicines impacts the guality of life of patients and carers. This enables 
them to make a fully informed decision on whether or not to recommend a new 
medicine. Patients, members of their families and carers can provide unique knowledge 
about what it's like to live with a condition. They can explain advantages and 
disadvantages of medicines that may not be available in the published literature or 
quality of life measures. We work in partnership with patient groups to gather this 
information through our patient group submissions. 

On a separate page the SMC uses the term 'public', stating 'the Public Involvement Network 
(PIN) Advisory Group helps SMC to continuously improve how we involve patients, carers and 
members of the public in our work' 37 . 

'Participation' is often used to mean 'involvement' yet there is potential for confusion as 
'research participants' or people 'participating in research' has a very specific meaning. 


34 http://web.archive.org/web/201 51 1 20090945/https://www. nice. ora.uk/about/nice- 
communities/public-involvement 

35 http://web.archive.org/web/201 51 1 24003854/https://www.cadth.ca/provide-input 

36 

http://web.archive.org/web/2Q1 51 1 2400491 8/https://www.scottishmedicines. ora.uk/Public Involvem 
ent/Public Involvement 

37 

http://web.archive.ora/web/201 51 1 24005229/https://www.scottishmedicines. ora.uk/Public Involvem 
ent/Public Involvement Network Advisory Group 
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Similarly, organisations often say 'engagement' when they mean involvement and vice versa. 
Accordingly, INVOLVE (UK) provides this helpful articulation of the terms: 

'Researchers and others use different words to describe public involvement, for 
example words such as engagement and participation. When INVOLVE uses the term 
'public involvement' we are not referring to researchers raising awareness of research, 
sharing knowledge or engaging and creating a dialogue with the public. We are also 
not referring to the recruitment of patients or members of the public as participants in 
research. However, these different activities - involvement, engagement and 
participation - are often linked and, although they are distinct, can complement each 
other. For example, the public can and do play a valuable role in advising on 
recruitment of patients as participants and on ways of engaging with the public' 38 . 

Who are 'the public'? 

In the paper 'Role of patient and public participation in health technology assessment and 
coverage decisions', Menon writes: 

'Published work on public involvement in healthcare has defined a number of 'publics'. 
They include patients and users of health services, caregivers, taxpayers and 
representatives of these publics, such as elected officials and patient advocacy 
groups. Such groups are not always mutually exclusive, with individuals belonging to 
more than one at any given time. This article considers two specific groups: citizens, 
representing society (who will be referred to as the 'public') and patients, as users of 
the health system. Although patients are also members of the public, for the purpose 
of this paper, their 'engagement' in HTA will focus on their experience living with a 
condition requiring interactions with the health system' (Menon, A037) 39 

Menon goes on to state: 

'Patients offer important insights into the value of a technology through first-hand 
experience with the condition in which the technology is used. By contrast, the public, 
representing taxpayers, may provide input into the 'relative value’ of a technology when 
compared with other services under consideration for funding at the same time' 
(Menon, A037) 40 

The use of the term 'public' reflects a shift in language (and perhaps attitudes) both 
domestically and internationally. 

The Health Research Authority in England provides a helpful definition of the word public, 
stating; 


'When we use this term, public means patients, potential patients or members of the 
public including those with known genetic dispositions, carers and people who use 


38 http://web.archive.org/web/201 51 1 26053458/http://www.invo. org.uk/find-out-more/what-is-public- 
involvement-in-research-2/ 

39 https://drive.aooale.eom/file/d/OBwQTUaazNAv5UEROckEwbl96VEE/ 

40 https://drive.aooale.eom/file/d/OBwQTUaazNAv5UEROckEwbl96VEE/ 


health and social care services as well as people from organisations that represent 
people who use health and social care services ' 41 


Similarly, INVOLVE (a UK organisation which is funded by the National Institute for Health 
Research, to support active public involvement in NHS, public health and social care research) 
states: 


'When using the term 'public' we include patients, potential patients, carers and people 
who use health and social care services as well as people from organisations that 
represent people who use services. Whilst all of us are actual, former or indeed 
potential users of health and social care services, there is an important distinction to 
be made between the perspectives of the public and the perspectives of people who 
have a professional role in health and social care services' 42 . 

The previous two definitions are analogous to one used by the National Health and Medical 
Research Council (NHMRC) in the 'Draft Statement on Consumer Involvement in Health and 
Medical Research', which defined the word consumer as: 

'patients and potential patients, research participants, carers, consumer organisations 
and members of the public ' 43 

Similarly, the 'Public Involvement Programme' (PIP) is a team at the National Institute for 
Health and Care Excellence (NICE) that ‘develops and supports patient, carer and public 
involvement '. 44 

Use in this document 

The term 'public' has been used in this document, reflecting the language used in the 
recommendations section of the Senate Community Affairs References Committee report 
'The availability of new, innovative and specialist cancer drugs in Australia', where it states a 
need to improve ways of 'incorporating public perspectives on overarching moral, ethical and 
opportunity cost considerations into PBAC decision making processes ' 45 

'Public involvement' will be used throughout, to be understood to also include 'consumer, 
patient and carer involvement'. 

Please see the next section 'Mapping the terms' for a visual explanation of these concepts. 


41 http://www.hra.nhs.uk/documents/2013/10/hra-public-involvement-strategv-circulation-september- 
2013.pdf 

42 http://web.archive.org/web/201 51 1 26053458/http://www. invo.org. uk/find-out-more/what-is-public- 
involvement-in-research-2/ 

43 

http://consultations.nhmrc.aov.au/files/consultations/drafts/draftconsstatementconsultationversion1 

40807.pdf 

44 https://www.nice.ora.uk/about/nice-communities/public-involvement/public-involvement- 
programme 

45 

http://www.aph.aov.au/Parliamentarv Business/Committees/Senate/Communitv Affairs/Cancer Drug 
s/~/media/Committees/clac ctte/Cancer Drugs/Report/report.pdf 
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Mapping the definitions: Diagram of engagement, 
participation and involvement in research 

This diagram attempts to map the various definitions and terms explored in this report, to aid 
a discussion about the most helpful terms to use at which stages of the MBS Review. This 
diagram is adapted from a resource created by Jack Nunn for Macmillan Cancer Support 46 



Engage 


Communicating and informing people 




Use services 
or items 

As a patient, service user, 
consumer or person 


General 

Public 

Everyone is a potential 
patient, consumer or 
service user 


Get involved 

Help prioritise issues, design 
services, recruit to and carry 
out research, evaluate 
services or items, 
disseminate and translate 
findings 


http://web.archive.org/web/201 51 1 20021 821 /http://learnzone. org.uk/courses/course. php?id=1 09 


Detailed appraisal of five relevant models 


This sections examines five internationally relevant models in detail, summarising the design 
and where appropriate identifying key elements and limitations. Rather than being a selection 
of 'key models' these five models represent the only directly relevant models found during the 
literature search. 

After the detailed appraisal of these models is a short section summarising two other 
significant models which contain useful elements. 

Appraisal of 'Role of patient and public participation 
in Health Technology Assessment and coverage 
decisions' (Menon, A037 47 ) 

This diagram displays the main phases within the Health Technology Assessment process. 
This process begins with 'Innovation' at the initial phase and 'Abandonment' or 'Obsolescence' 
at its conclusion. 


Pradmcal and efnical tnals 




Mfuson 


Post -approval evaluation 


Technology pokey sequence 


1 1 1 1 1 . 

Research and 
development 

poke**' 

Regulatory approval 
process 

Coverage approval/ 
reimbursement process 9 

Routine use m 
healthcare systom 
(clinical pokcws) 

Abandonment 
or obsotosconco 
(tasmvestmont 


pekoes) 


1 

Regulatory pokey 


Public access 


J L 


Coverago pokey 


Figure 1. Relationship between the life cycle of a technology and a technology policy. 

'Points at which the public has been involved in health technology policy 


This model raises the guestion about where in this process the public are currently involved, 
and where involvement could be developed or improved. 


47 https://drive.aooale.com/open?id=OBwQTUaazNAv5UEROckEwbl96VEE 
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Summary of differences between leading countries 


Based upon the research of Menon and Stafinski, the following have been identified as current 
mechanisms employed by various countries, states or provinces: 


Phase 

Process 

Identification of technologies 

• 64% of national processes and 75% of state processes 
involved the public in identifying technologies for 
assessment/potential coverage. 

• In general the referral process involved completion of 
a downloadable 'application form' located on the 
respective organizations website. 

Selection of technologies for 
assessment (priority setting) 

• With two exceptions (NICE, UK and Centers for 
Medicare and Medicaid Services, CMS) none of the 
national, state or provincial processes sought input 
from patients or the public in the selection of 
technologies for assessment 

Conduct of HTA (including defining 
the scope of the HTA) 

• All but one national process elicited the views of 
patients and/or the public during the actual 
assessment. 

• Two processes included patient representatives on 
committees tasked with defining the scope of 
assessment (Australia and France). 

• One sought input into the scope from patient/carer 
organizations (UK) and another from anyone 
(Germany). 

• Several processes invited anyone (Germany) or patient 
representatives (France & UK) to comment on draft 
protocols. 

• Patients/carers were able to submit information to the 
body responsible for preparing the assessment 
directly (Germany), through patient/ carer organization 
'group' submission (Canada, France, UK) or through 
the appointment of a patient representative to the 
advisory group overseeing the assessment (Australia). 

• Two of the processes accepted information from 
anyone (Germany and USA). 

• One process facilitated input from patients through 
dedicated patient and public involvement 
subcommittee, which also included three members of 
the general public (Scotland). 

• In one process, submissions were accepted only if 
requested by the decision making committee (New 
Zealand). 



• 

Review of the HTA results and 
formulation of recommendations 

• Committee membership-appointment of at least one 
patient representative to the committee responsible 
for making funding recommendations (Australia, 
France, Germany [in the German committee the 
patient is a non-voting member]). 

• Presentation of written or oral testaments from patients 
(consumer impact statements) - either by committee 
invitation (Australia, The Netherlands) or at the request 
of the patients/carer organization (UK). 

• Opportunities to review HTA report and draft 
recommendations- (France and UK). Patient 
representatives included on committees were typically 
recruited through patient/carer organizations. In one 
process, the role of the public in this step was limited 
to committee membership (Canada) or commenting 
upon draft recommendations (Germany and the USA. 
Finally, several processes held committee meetings 
partly in public (UK). 

Implementation of 
recommendations/decisions 

• Generally, public or patient involvement in the 
implementation of recommendations was limited to 
the launch of appeals against recommendations. With 
two exceptions (UK and USA), only those who 
represented the 'applicant' could initiate the appeal 
(Australia). 

Dissemination of decision and HTA 
findings 

• The majority of national and provincial/ state level 
processes posted reports and decisions/ 
recommendations on organizational websites. 
However, neither stakeholder group appeared to be 
actively involved in the dissemination of such 
information. 


Australia: Detailed analysis 
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Key elements 




"Health technology policy making cannot be reduced to a technical exercise, free from 
values or ethical judgements." 

Menon highlights the need for qualitative methodologies, to compliment quantitative ones: 

'Well-accepted methods to collect clinical trial data, scales to assess the quality of 
clinical trials and guidelines for the conduct of economic evaluations already exist and 
are adhered to by most HTA producers. However, patients' experiential information 
will, at least to some extent, be qualitative, requiring different methodological 
approaches (e.g., focus groups and interviews). Although the published literature 
encourages the collection and use of such information, it offers little guidance on 
rigorous approaches to accomplish this, given the time and resource constraints faced 
by most HTA bodies.' 

Menon identifies keys stages of the HTA process, explaining the importance of public 
involvement at each stage: 

Completing a HTA involves several steps, including: identifying the 'right' questions; 
defining the appropriate outcomes and other measures; collecting, analyzing and 
synthesizing information; and writing a report intended for decision-maker. 

Menon continues: 

'Patients should be involved in all of these steps to ensure that HTA adopts a health 
condition perspective, rather than a technology perspective. It is argued that their input will 
identify questions to be addressed that differ from those typically formulated by HTA 
agencies, governments or payers. Patients' views on what constitutes 'value' may not be 
the same as those of clinicians or trialists. This may lead to the identification and selection 
of outcome measures that do not capture critical aspects of 'benefit' to patients (e.g. 
preferences regarding the management of their condition). Through active involvement of 
patients in this step, such concerns could be alleviated' 

Menon cites the European Network for HTA, which defines HTA as 'a multidisciplinary process 
that summarizes information about the medical, social, economic and ethical issues related to 
a health technology in a systematic, transparent, unbiased, robust manner'. 

Menon writes about the role of the 'public' and 'patients'. Please note that this content is 
repeated and highlighted in the section 'Exploring definitions and terms'. 

• 'Published work on public involvement in healthcare has defined a number of "publics”. 
They include patients and users of health services, caregivers, taxpayers and 
representatives of these publics, such as elected officials and patient advocacy groups. 
Such groups are not always mutually exclusive, with individuals belonging to more than 
one at any given time. This article considers two specific groups: citizens, representing 
society (who will be referred to as the 'public') and patients, as users of the health system. 
Although patients are also members of the public, for the purpose of this paper, their 
'engagement' in HTA will focus on their experience living with a condition requiring 
interactions with the health system'. 
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• 'Patients offer important insights into the value of a technology through first-hand 
experience with the condition in which the technology is used. By contrast, the public, 
representing taxpayers, may provide input into the 'relative value' of a technology when 
compared with other services under consideration for funding at the same time'. 

Menon discusses the importance of public involvement not only to the public, but also to 
uphold the validity of the HTA process (and thus funding) itself: 

'Unless greater effort is made to involve patients and citizens in determining priorities, 
in evaluating the efficacy and cost-effectiveness of healthcare interventions, and even 
more importantly, in using the results of these evaluations to make informed choices, 
HTA will have little chance of achieving its goals. It will also be hard to sustain public 
support for funding HTAs if the public remains ignorant of its importance and relevance 
to them. It is assumed that through active patient involvement in identifying and 
selecting technologies, those that are 'really important' (i.e., those that matter to 
people suffering an illness and therefore impact health) will be chosen for 
assessment. This could, in turn, change the perception of HTA from a 'gatekeeper' to an 
'enabler' of effective new technologies. At present, criteria for identifying and selecting 
technologies for assessment are typically developed or adopted and used by HTA 
agencies without the direct involvement of patients or the public.' 

Menon summarises the HTA process in the following way: 

• Identification of technologies 

• Selection of technologies for assessment (priority setting) 

• Conduct of HTA (including defining the scope of the HTA) 

• Review of HTA results and formulation of recommendations (e.g., fund, fund with 
conditions, provide interim funding and disinvest) 

• Implementation of recommendations/decisions 

• Dissemination of decision and HTA findings 

Evaluation framework 

Menon offers a framework for analysing public involvement in HTA. This framework 
'recognizes the various facets of possible public engagement in HTA as three domains': 

• the policy making domain 

• the organizational domain 

• the research domain. 

Each domain 'comprises a series of activities, including priority setting for assessments, 
collection of evidence and synthesizing data. A dedicated literature search for information 
describing its validation or application was conducted, but no studies were found.' 

Key Issues iden tified by Menon 

• Patients and members of the public may have different roles and should be 
considered separately when they are to be involved in health technology assessment. 

• There are methodological challenges with respect to how information on the 'patient 
experience' should be collected and synthesized. 

• Since many patient organizations receive funding from industry sources, there is 
concern about the objectivity of the information these organizations provide. 


• Since not all patients share the same views it is important to canvas multiple patients 
and patient organizations. 

• Despite a lack of evaluative evidence on the comparative merit of different approaches 
to engaging the public and patients, there seems to be a general consensus that 
obtaining their perspectives is important. 

Limitations 

The 'decentralized nature' of health-care systems and the international variation makes 
meaningful detailed comparisons challenging. 

Although a 'one-size-fits-all approach to HTA may be appropriate for certain technologies', 
location specific information and data is playing an increasingly important role in decision 
making, for example, 'patient preference, resource implications, ethical considerations, 
implementation issues'. 

This increasingly localised system limits the way in which meaningful comparisons can be 
made between systems, although it is not a barrier to learning from the different kinds of 
involvement. 


This document was created by Jack Nunn for the Consumer Health Forum on 1 1 October 2016. To give feedback, send an email or contact 
fSJackNunn . 


35 


Appraisal of 'A Pilot Study to Identify Areas for 
Further Improvements in Patient and Public 
Involvement in Health Technology Assessments for 
Medicines' (Messina, A022) 48 

The objective of this paper was to 'conduct a pilot study to form the basis of future research 
and to gain insight into how to practically and meaningfully advance patient and public input in 
HTAs for medicines' (Messina, A022) 49 . 

The authors 'apply learning from health systems in Australia, Canada, and the UK to focus on 
the practical aspects of engagement'. Australia's 'patient and public participation within the 
HTA process is not as systematic as the English model; however, public involvement in the 
decision making process can take on three distinct forms: committee member representation, 
brief patient group submissions via a template process and, most recently, more detailed 
consumer impact statements'. 

Consumer impact statements 'are a PBAC pilot program in Australia that utilize qualitative 
research to examine what it is like to live with a particular disease or condition, for which a 
new treatment will be considered'. The authors state that: 

'While these documents have been cited as useful in the medicine HTA process, to 
date they are only used for rare or poorly understood conditions where the committee 
will reguire more evidence on the patient's guality of life and experience living with the 
condition'(Messina, A022) 50 . 

The pilot study identified three key areas for 'further advancement' in the field of 'public input' 
in the HTA process: 

Advancement 1: Industry 

• Industry could help bring patient perspectives into the HTA process through incorporating 
patient experiences early in the drug development and by including gualitative research on 
patient experiences in HTA dossiers. 

• This will place a greater importance on the patient as an 'adviser'. 

• The patient's perspective can provide rich and valuable data when incorporated early on in 
the technology development stage, allowing patients to voice their opinions and exert 
some influences early on in the process. However, the legal and regulatory constraints 
that may exist need to be considered and navigated. 


48 https://drive.qooqle.com/open7icHOBwQTUqazNAv5WEZNN2lveilOQkU 

49 https://drive.aooqle.com/open7icHOBwQTUqazNAv5WEZNN2lveilOQkU 

50 https://drive.qooqle.com/open?id=OBwQTUqazNAv5WEZNN2lveilOQkU 


• There is an imperative need to integrate qualitative methods in the HTA process so that 
rich accounts of the patient experience are captured. Scientific quantitative methods alone 
fall short of capturing integral patient information, perceptions and attitudes. 

• Industry need to give careful consideration to what it is like living with a particular disease 
and how any new technology might improve patient experience. 


Advancement 2: Patient Advocacy Groups 

• Recognizing and supporting the role of patient advocacy groups and making use of their 
access to the genuine patient perspective and experience of living with the condition in 
question. 

• Patient groups should be properly supported along the way so that they can make a 
meaningful contribution to the process. 

• Patient groups should be given early notice of upcoming product reviews to ensure they 
have adequate time to organize a patient submission. For example, in Australia, patient 
advocacy groups have a 2-week window to make submissions to the PBAC. Industry need 
to ensure that timelines do not pose a barrier to patient engagement. 

• Patients feel intimidated by the process and need to be adequately supported so that they 
don't feel like outsiders to the HTA process. 

• The paper also outlines a number of recommendations for learning and development 
which can be found in the section 'Creating opportunities for learning, education, training 
and development'. 

Advancement 3: HTA Process Developments 

• The continuous development of HTA systems and processes to better facilitate 
involvement, increasing transparency and feedback, exploring new options for reaching 
patients and focusing on creating an active and informed consumer. 

• Adequate mechanisms for feedback in the process were noted as suitable options for 
enhancing consultations in HTA's. 

• Goals, priorities, and outcomes for the health system should be discussed in open forums 
with wider health stakeholders. 

• Education and training to patients and the public so they can be active and informed 
consumers who are health literate and can contribute to the development and progression 
of medicine HTA's. An educated public can provide quality information on citizen's values, 
needs, judgments, prioritization and technology preferences. 

• A proper feedback mechanism needs to be developed in order to address how patient 
input was utilized in the HTA process. 
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Framework for increasing patient and public engagement in medicine 
health technology assessments 

The following framework outlines 'ways of increasing patient and public engagement in 
medicine health technology assessments'. It was formulated based on the pilot study 
conducted by Messina and Grainger, which utilized semi- structured interviews with agencies, 
experts and patient and advocacy groups in Australia, Canada and the UK. 


Table I. Framework lor increasing patient and public engagement 
nr medicine bealth technology assessments 

HTA advancement areas 

1. Industry 

Qualitatfte research by the sponsor 
Industry focus on OOL 

Eariier consideration ot Ihe imped of 'living wilh Ihe disease 1 
Inclusion d the patient's perspective in sponsor dossiers 
£.. Patient advocacy groups 

Third-party facilitated discussion wir patient advocates 

Adding context to HTA data including OOL and clinical outcome 
measures 

Timely notification of new products 

Time 1o commission research and ensure support for patient 
engagement 

Delivery of patient submissions and re miration of advocates to 
represent patients 

3. HTA process development 

Wider discussion of the role ol HTA in Ihe health system 

Educate patient advocacy gimps end HTA conmittees on ihe HTA 
process 

Consideration of new media options 

Develcpmenl of new methods tc reach patienlsflhe public 

Increased process transparency 

Document meeting outcomes mcludng ihe value ol patient input 

Formal audit ol Ihe process 

Understanding values and priorities 

Citizens' juries 

Health literacy 

Promotion of active and informed consumers 

HTA= health technology assessment: Q0L = quai1y of file. 


Significant quotations from interviewees 


These direct quotations from respondents 'serve to illustrate larger themes from the interview 
data collected in this study'. 


“ You should have meaningful involvement of 
patients throughout the process, on an equal 
footing to other members of an assessment 
group, and that your money should be invested 
in good qualitative research to determine [the] 
patient's perspective ... that has to happen very 
early, because qualitative research takes time 
... and so it needs to he identified early in the 
process and, indeed, from a pharmaceutical com- 
pany perspective, / think qualitative research 
should be happening earlier in drug develop- 
ment." (Agcncy/cxpcrt. the UK) 


"To start off, l think there needs to be more of 
an education about HTA and the process. It 
needs to be explained in such a way that people 
who don 7 have any prior experience can under- 
stand it." (Patient advocacy, the UK) 

"I think it's the time pressures - by the time 
people [patients] get around to thinking 7 want 
to put something in (a patient submission]' it’s 
almost too late ... it's gone ... so it isn 7 enough 
time." (Agcncy/cxpcrt. Australia) 


"Patients often bring a very unique perspective 
... and cut through to the real issues: ‘okay, this 
means if l gel this drug, this means l can afford 
to pay my mortgage and receive treatment that 
will keep me alive for another couple of years’" 
(Patient advocacy, Australia) 

“Patients are the ultimate end user of any 
technology and they can tell us more about 
living with the disease, actually using the 
technology, and any challenges they find, than 
anybody else can." (Agcncy/cxpcrt. the UK) 


"I would like some sort of [new improvement ], 
that patients or citizens would he equal partners 
with other groups that input on decision making, 
so their voice would be seen as an equal partner 
... with those other pieces of evidence." 
(Acadcmic/cxpcrt, Australia) 
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Identified mechanisms to improve public involvement in the HTA process 


The paper outlines the following areas for improvement and recommends mechanism for 
improving them. These areas and mechanisms are summarised in the table below. 


Area for improvement 

Mechanism 

Lack of education which leads to poorly 
produced patient submissions. 

Patient and advocacy groups to be 
supported by industry with education and 
training as to how to produce a good 
submission. 

Inadequate feedback and transparency 
channels 

A formal audit would provide a clear 
understanding of how patient input was 
utilised in HTA decision making. 

Lack of public or consumer involvement 

Considering new media options is likely to 
increase patient and public engagement as 
more people are turning to media for health 
information. It may also be possible to gain 
access to patient and public views through 
blogs and other social media platforms. 

Limited involvement in health policy decision 
making 

Open forums where patients and the public 
would be welcome to be part of wider health 
policy decision making. 


Key elements 


The pilot study identified three key areas for 'further advancement' in the field of 'public input' 
in the HTA process: 

Advancement 1 : industry could help bring the patient perspective into the HTA process 
through incorporating patient experiences early in the drug development process and by 
including qualitative research on patient experiences in HTA dossiers. In addition 'It would be 
helpful to the HTA process if patients' needs and views were considered in the design of 
phase III research trials conducted by industry'. 

Advancement 2: recognizing and supporting the role of patient advocacy groups, and making 
use of their access to the genuine patient perspective and experience of living with the 
condition in question. 

Advancement 3: the continuous development of HTA systems and processes to better 
facilitate involvement, increasing transparency and feedback, exploring new options for 
reaching patients, and focusing on creating an active and informed health consumer. 

The paper includes the following 'key points for decision makers': 

• The health technology assessment (HTA) of medicines should incorporate the 
patient's experience of what it is like living with the disease and views on the 
technology from the patient's perspective 

• Industry can assist in acquiring patient voices by including patient views early on in 
drug development 


• Increasing transparency and feedback, and exploring new options for reaching 
patients are needed 

The paper also notes that 'the exact point to seek input from patients and the public is 
controversial; however, it suggests that 'those involved within HTAs agree that patients/the 
public are usually involved too late in the process'. 

Limitations 

The exploratory nature of this pilot study uncovered 'various themes and advancements to the 
current HTA systems in Australia, Canada, and the UK'. The authors note that 'these findings 
may be subject to change if the same questions were asked to a larger sample': 

'While we would expect similar themes to emerge in a larger sample, there may be 
additional explanations and discussions of areas for improvement that materialize in a 
larger sample. Another limitation of this research is the selection of the three 
geographic case studies in which we sampled. Since experiences and knowledge are 
linked to each respondent's HTA system and country, the findings presented in this 
paper are most relevant to Australia, Canada, and the UK' 

The paper only examines models in the English speaking world, although 'the suggested areas 
of advancement certainly have applications to HTA systems across the globe'. 
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Appraisal of 'Involving patient in the early stages of 
health technology assessment (HTA): a study 
protocol' (Gagnon, A015) 51 

Model for HTA 

This model depicts interventions to promote patient involvement in the early stages of the 
HTA process, as 'in health technology assessment (HTA), public and patient involvement is 
also considered as a priority' (Gagnon, A01 5 52 ). The three levels of patient involvement are: 

1. Communication and information - Mechanisms are put in place to inform and 
raise awareness among patients. 

2 . Consultation- Mechanism are put in place to collect information or opinions 
from patients. 

3. Active participation- Mechanisms are put in place to actively involve patients as 
partners, which is considered the most advanced form of involvement. 

Here is an extract from the model (please note the authors intended the word 

'participate', not participle): 


51 https://drive.aooale.eom/file/d/0BwQTUaazNAv5b3huQ2JWS1 pwTm8 

52 https://drive.aooale.eom/file/d/0BwQTUgazNAv5b3huQ2JWS1 pwTm8 


r 


Level 1 -Information 


Mechanisms are put in 
place to inform and raise 
awareness among patients 

^ , i ^ 

r Level 2- Consultation^V 


1 


Mechanisms are put in 
place to collect 
information or opinions 
from patients ^ 


X 


Identify potential 
HTA topics in the 
field of cancer 




Level 3 - Participation 

Mechanisms are put in 
place to actively involve 
patients as partners 


T 


Revise 
vignettes 

Participe in 
priorisation 
committees 


Participate in 
working 
group for the 
assessment 
plan 


In addition to the three level, there are three stages in the HTA process outlined in this 
model: 


Stage 1 - Selection of evaluation topics which involves submitting and prioritising 
reguests. This is the early stage of the process, at which the authors identified a need 
for patients to be more actively involved in identifying topics 

Stage 2 - Evaluation, which involves developing an assessment plan, collecting 
evidence of the literature, analysing and synthesising results and finally discussing 
results and recommendations. 

Stage 3 diffusion - (often called dissemination), where the report is 'diffused' and 
shared with the public and other stakeholders. 
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HTA process 


r 

Selection of 
evaluation topics 

r N 

Evaluation 

r \ 

Diffusion 

r ' 

Submetting 

requests 

' ' 

Prioritizing 

requests 

S, , 

Developing an 
assessment 
plan 

s -> 

r Collecting ^ 
evidence 
(Literature 
or new 

L data) j 

r 

Analysing 

and 

syntheising 

results 

r > 

Discussing 
report and 
recomman 
dations 
- -> 

f \ 

Diffusing 

report 

^ J 


Key elements 

• Recommends that a third of the participants involved in the deliberation process of 
prioritizations topics for HTA must be patients. 

• Advocates patient involvement in all stages of the HTA process as patient involvement in 
early stages make assessments 'more relevant and acceptable' (Gagnon, A01 5 53 ). 

• Claims that methods of involvement 'that involve face-to face interaction between 
participants from the public and decision makers also generate greater participant 
satisfaction, both in terms of process and results' (Gagnon, A01 5 54 ). 

• Based on learning from the UK, the authors suggest patients should 'receive training in 
HTA and evidence-informed healthcare before the prioritisation meetings' (Gagnon, 
A015 55 ). 

• Patients are involved in the consultation and reviewing process of a potential HTA topic, as 
well as development of the assessment plan. 

Limitations 

Provides limited information regarding: 

• The evaluation stage, for example committees involved in consulting and reviewing 
evidence. 

• Measuring the impact of impact of patient involvement, for example, how effective 
were the strategies to involve the public and did the involvement improve the process 
and outcomes. 

This paper cites an alternative model for evaluating participation and involvement, 'Public 
Participation Methods: A Framework for Evaluation' (Rowe, A051) 56 . 


53 https://drive.aooale.eom/file/d/0BwQTUaazNAv5b3huQ2JWS1 pwTm8 

54 https://drive.aooale.eom/file/d/0BwQTUgazNAv5b3huQ2JWS1 pwTm8 

55 https://drive.qooqle.eom/file/d/0BwQTUqazNAv5b3huQ2JWS1 pwTm8 


Appraisal of 'Public Participation Methods: A 
Framework for Evaluation' (Rowe, A051) 57 

Rowe outlines a number of 'formalised public participation models' (Rowe, A051 ) 58 : 


Table 1. A Number of the Most Formalized Public Participation Methods 


Participation 

Method 

Nature of Participants 

Time Scale/Duration 

Characteristics/Mechanism 

Examples/References 

Referenda 

Potentially all members of 
national or local population; 
realistically, a significant 
proportion of these. 

Vote cast at single point in 
time. 

Vote is usually choice of one Biotechnology in Switzerland 
of two options. All participants (Buchmann 1995); waste 
have equal influence. Final repository in Sweden 
outcome is binding. (af Wahlberg 1 997). 

Public 

hearings/ 

inquiries 

Interested citizens, limited in 
number by size of venue. 
True participants are experts 
and politicians making 
presentations. 

May last many weeks/ 
months, even years. 
Usually held during week- 
days/working hours. 

Entails presentations by Frequent mechanism in, for 

agencies regarding plans in example, United States 
open forum. Public may voice (Fiorino 1990), Australia 
opinions but have no direct (Davison, Barnes, and Schibeci 
impact on recommendation. 1997); review by Middendorf 
and Busch (1997). 

Public opinion 
surveys 

Large sample (e.g., 100s or 
1,000s), usually representa- 
tive of the population seg- 
ments of interest. 

Single event, usually lasting 
no more than several 
minutes. 

Often enacted through writ- 
ten questionnaire or tele- 
phone survey. May involve 
variety of questions. Used for 
information gathering. 

Radioactive sites in United 
States (Feldman and Hanahan 
1996); genetically modified 
food in the United Kingdom 
(Vidal 1998); biotech surveys 
(Davison, Barnes, and 
Schibeci 1997). 

Negotiated 
rule making 

Small number of represen- 
tatives of stakeholder 
groups (may include public 
representatives). 

Uncertain: strict deadline 
usually set: days/weeks/ 
months. 

Working committee of stake- 
holder representatives (and 
from sponsor). Consensus 
required on specific question 
(usually a regulation). 

Used by U.S. Environmental 
Protection Agency (Hanson 
1984); method discussed by 
Susskind and McMahon (1985) 
and Fiorino (1990). 


56 

http://web.archive.org/web/201 51 1 1 8033233/http://www.en.ipea.aov.br/participacao/imaaes/pdfs/pa 
rticipacao/2000%20public%20participation%20methods.pdf 

57 

http://web.archive.org/web/201 51 1 1 8033233/http://www.en.ipea.aov.br/participacao/imaaes/pdfs/pa 
rticipacao/2000%20public%20participation%20methods.pdf 

58 

http://web.archive.org/web/201 51 1 1 8033233/http://www.en.ipea.aov.br/participacao/imaaes/pdfs/pa 
rticipacao/2000°/o20public%20participation%20nnethods.pdf 
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Consensus 

conference 

Generally, ten to sixteen 
members of public (with no 
knowledge on topic) selected 
by steering committee as 
"representative" of the 
general public. 

Preparatory demonstrations 
and lectures (etc.) to inform 
panelists about topic, then 
three-day conference. 

Citizens' 

jury/panel 

Generally, twelve to twenty 
members of public selected 
by stakeholder panel to be 
roughly representative of 
the local population. 

Not precise but generally 
involve meetings over a few 
days (e.g., four to ten). 

Citizen/public 

advisory 

committee 

Small group selected by 
sponsor to represent views 
of various groups or com- 
munities (may not comprise 
members of true public). 

Takes place over an 
extended period of time. 

Focus groups 

Small group of five to twelve 
selected to be representative 
of public; several groups may 
be used for one project 
(comprising members 
of subgroups). 

Single meeting, usually 
up to two hours. 


Lay panel with independent 
facilitator questions expert 
witnesses chosen by stake- 
holder panel. Meetings open 
to wider public. Conclusions 
on key questions made via 
report or press conference. 

Lay panel with independent 
facilitator questions expert 
witnesses chosen by stake- 
holder panel. Meetings not 
generally open. Conclusions 
on key questions made via 
report or press conference. 

Group convened by sponsor 
to examine some significant 
issue, interaction with 
industry representatives. 


Free discussion on general 
topic with video/tape record- 
ing and little input/direction 
from facilitator. Used to 
assess opinions/attitudes. 


Used in Denmark and Nether- 
lands on topics from food 
irradiation to air pollution (Joss 
and Durant 1994; Grundahl 
1995); also used in United 
Kingdom on plant biotech- 
nology (Ellahi 1995). 

Examples in Germany, United 
States, and United Kingdom 
(e g., Crosby, Kelly, and 
Schaefer 1986; Coote, Kendall, 
and Stewart 1994; Lenaghan, 
New, and Mitchell 1996). 


Particularly evident in United 
States, for example, in cleanup 
of waste sites (Lynn and 
Busenberg 1995; Perhac 
1998); see Creighton (1993) 
for guidelines. 

Guidelines from Morgan 
(1993); U.K. example to 
assess food risk (Fife-Schaw 
and Rowe 1995). 


Evaluating the 'construction' and 'implementation' stages 

In addition to outlining 'formalised public participation models', Rowe also proposes the 
following framework for evaluating the 'effectiveness' of the various models of participation. 

Evaluation criteria are 'divided into acceptance criteria, which are related to the effective 
construction and implementation of a procedure, and process criteria which are related to the 
potential public acceptance of a procedure' (Rowe, A051) 59 . 

This split is justified by Rowe in the following way: 

'If a procedure is effectively constituted but perceived by the public to be in some 
sense unfair or undemocratic, then the procedure may fail in alleviating public 
concerns. On the other hand, if a procedure and its recommendations are accepted by 
the public but the ultimate decision is attained in an ineffective manner, then its 
implementation could prove objectively damaging for sponsors and public' (Rowe, 
A051) 60 . 

The 'acceptance criteria' are: 

Criterion of representativeness: The public participants should comprise a broadly 
representative sample of the population of the affected public. 

Criterion of independence: The participation process should be conducted in an independent, 
unbiased way. 


59 https://drive.aooale.eom/file/d/0B7-zEXxcoXA4QU9hWEVNNnl0Y0U/view 

60 https://drive.aooale.eom/file/d/0B7-zEXxcoXA4QU9hWEVNNnl0Y0U/view 


Criterion of early involvement: The public should be involved as early as possible in the process 
as soon as value judgments become salient. 

Criterion of influence: The output of the procedure should have a genuine impact on policy. 

Criterion of transparency: The process should be transparent so that the public can see what 
is going on and how decisions are being made. 

The 'process criteria' are: 

Criterion of resource accessibility: Public participants should have access to the appropriate 
resources to enable them to successfully fulfil their brief. 

Criterion of task definition: The nature and scope of the participation task should be clearly 
defined. 

Criterion of structured decision making: The participation exercise should use/provide 
appropriate mechanisms for structuring and displaying the decision-making process. 

Criterion of cost-effectiveness: The procedure should in some sense be cost-effective. 
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Results of the evaluation 


Rowe provides a detailed commentary about each method and summarises the findings in 
the following table: 



Referenda 

Public 

Hearings 

Public 

Opinion 

Survey 

Negotiated 

Rule 

Making 

Consensus 

Conference 

Citizens' 

Jury/Panel 

Citizen 

Advisory 

Committee 

Focus 

Groups 

Acceptance criteria 









Representativeness 
of participants 

High (assum- 
ing full turn- 
out at poll) 

Low 

Generally 

high 

Low 

Moderate Moderate 

(limited by (limited by 

small sample) small sample: 

Moderate 1 
to low 

1 

Moderate 
(limited by 
small sample) 

Independence of 
true participants 

High 

Generally 

low 

High 

Moderate 

High 

High 

Moderate 
(often relation 
to sponsor) 

High 

Early involvement? 

Variable 

Variable 

Potentially 

high 

Variable 

Potentially 

high 

Potentially 

high 

Variable but 
may be high 

Potentially 

high 

Influence on final 
policy 

High 

Moderate 

Indirect and 
difficult to 
determine 

High 

Variable 
but not 
guaranteed 

Variable 
but not 
guaranteed 

Variable 
but not 
guaranteed 

Liable to be 
indirect 

Transparency of 
process to the public 

High 

Moderate 

Moderate 

Low 

High 

Moderate 

Variable but 
often low 

Low 


Referenda 

Public 

Hearings 

Public 

Opinion 

Survey 

Negotiated 

Rule 

Making 

Consensus 

Conference 

Citizens' 

Jury/Panel 

Citizen 

Advisory 

Committee 

Focus 

Groups 

Process criteria 









Resource accessibility 

Low 

Low- 

moderate 

Low 

High 

High 

High 

Variable 

Low 

Task definition 

High 

Generally 

high 

Low 

High 

Generally 

high 

Generally 

high 

Variable but 
may be high 

Variable but 
may be high 

Structured decision 
making 

Low 

Low 

Low 

Moderate 

Moderate 
(influence of 
facilitator) 

Potentially 

high 

Variable 
(influence of 
facilitator) 

Low 

Cost-effectiveness 

Variable/low 

Low 

Potentially 

high 

Potentially 

high 

Moderate 
to high 

Moderate 
to high 

Variable 

Potentially 

high 


Key elements 


Rowe proposes a framework for evaluating the 'effectiveness' of the various models of 
participation. 

The evaluation criteria are divided into: 

• Acceptance criteria - related to the effective construction and implementation of a 
procedure 

• Process criteria - related to the potential public acceptance of a procedure 
Rowe provides a detailed commentary about each method and summarises the findings. 

Limitations 

Rowe provides limited information regarding practical implementation of certain stages such 
as: 

• Resource implications not addressed- although time and scale are addressed, the 
model doesn't compare likely resource implications such as budget. 'Cost- 
effectiveness' is judged, but it doesn't include staff time or expertise. 

• Impact assessment - although the evaluation process for each stage is clearly 
articulated, there is no clear way described to measure the impact (real or perceived) 
of the public involvement proposed. 
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Appraisal of National Institute for Health and Care 
Excellence (NICE) process (England and Wales) 

This section is a summary and appraisal of the 'Public involvement programme' (PIP) during 
the NICE process HTA process, published in September 201 4 61 . In the interests of brevity, this 
report will not summarise the extensive appraisal process which can be read in full on the 
NICE website 62 

NICE summarises their patient and public involvement as being based on the following two 
principles 63 : 

• that lay people, and organisations representing their interests, have opportunities to 
contribute to developing NICE guidance, advice and quality standards, and support 
their implementation, and 

• that, because of this contribution, our guidance and other products have a greater 
focus and relevance for the people most directly affected by our recommendations 

Please note, NICE divides the appraisal process into Single Technology Assessment (ST A) 
and Multiple Technology Assessment (MTA) which, although similar processes, have slight 
variations in how the public are involved. 

Public involvement programme' (PIP) 

NICE has a 'Public involvement programme' (PIP) which 'supports and develops public 
involvement across NICE'S work programme’ 64 NICE 'offers support throughout the appraisal 
process to patient and carer consultee organisations and patient experts' 65 . During the 
appraisal process: 

'A PIP Public Involvement Adviser is assigned to each appraisal and supports patient 
and carer consultee organisations, their representatives, and individual patients or 
carers throughout the appraisal. The PIP public involvement adviser also supports the 


61 http://web.archive.org/web/201 51 1 20063506/https://www. nice, org.uk/article/pma1 9/chapter/3- 
The-appraisal-process 

62 http://web.archive.org/web/201 51 1 20063506/https://www. nice.org. uk/article/pmql 9/chapter/3- 
The-appraisal-process#sta-initiation-and-evidence-submission 

63 http://web.archive.org/web/201 50304054755/http://www. nice. orq.uk/about/nice- 
communities/public-involvement/patient-and-public-involvement-policv 

64 

http://web.archive.org/web/201 51 1 20082306/https://www.nice. org.uk/article/pma1 9/chapter/qlossar 

y 

65 http://web.archive.org/web/201 51 1 20063506/https://www. nice. org.uk/article/pma1 9/chapter/3- 
The-appraisal-process#sta-initiation-and-evidence-submission 


lay members of the Appraisal Committees and supplies the patient and carer group 
information for the 'Information for the public'.' 66 

NICE has a standing Advisory Committee with people representing patient and carer 
organisations and lay members. 

Patients Involved In NICE (PIN) 

Patients Involved in NICE (PIN) is a coalition of over 80 patient organisations and 'is 
committed to enabling patient groups to engage productively with NICE' 67 . It is independent 
from NICE and the pharmaceutical industry, and members: 

'use their combined knowledge, experience and direct contact with patients from a 
wide range of conditions, to ensure NICE puts patients, carers, and patient groups at 
the centre of all of its work. They act as a critical friend and a respected and equal 
partner in developing and shaping aspects of NICE'S work. They provide a forum for 
enabling patient groups to engage with NICE, working alongside NICE'S Public 
Involvement Programme.' 68 

Citizens Council 

NICE also has a 'Citizens Council' which 'is a panel of 30 members of the public that largely 
reflect the demographic characteristics of the UK. Councillors are recruited by an independent 
organisation and serve for up to three years' 69 . 

The Citizens Council 'is one of the few examples of broader public engagement (as opposed 
to specific patients or patient advocates)' (Messina, A022) 70 . 

The Citizens Council 'provides NICE with a public perspective on overarching moral and 
ethical issues that NICE has to take account of when producing guidance. The Council's 
recommendations and conclusions are incorporated into a document called 'Social Value 
Judgements' and, where appropriate, into NICE'S methodology' 71 . 

Operation of the Citizens Council 

• Members meet once a year for 2 days at a time and their discussions are arranged 
and run by independent facilitators. 

• The meetings are open to public observers. 


66 

http://web.archive.org/web/201 51 1 20082306/https://www.nice. ora.uk/article/pmal 9/chapter/alossar 

y 

67 http://web.archive.org/web/201 51 1 20090653/http://www. nice. ora.uk/about/nice- 
communities/public-involvement/pin 

68 http://web.archive.org/web/201 51 1 20090653/http://www. nice. ora.uk/about/nice- 
communities/public-involvement/pin 

69 http://web.archive.org/web/201 5031 01 1 3938/http://www.nice. ora.uk/aet-involved/citizens-council 

70 https://drive.aooale.com/open?id=OBwQTUaazNAv5WEZNN2lvejlOQkU 

71 http://web.archive.org/web/201 5031 01 1 3938/http://www. nice. ora.uk/aet-involved/citizens-council 
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During the meetings, Council members listen to different views from experts on a topic 
and undertake exercises which allow them to examine the issues in detail and 
thoroughly discuss their own views. 

The members' views and conclusions are captured by an independent rapporteur and 
the report is circulated to members for comment and amendment before finalising. 
After a meeting, the report is made available for public comment. 

A summary of these comments along with the report are then presented to NICE'S 
board for discussion. 



Significant reports by the Citizens Council 

The Citizens Council produced a report in which they pose the question 'What are the societal 
values that need to be considered when making decisions about trade-offs between equity 
and efficiency?' 72 . The document concluded that 'balancing equity and efficiency is an 
extremely difficult exercise and that there will never be a perfect balance' 73 . The word they 
chose to summarise the values 'underlying' their work was 'education' as: 

'it 'reflects underlying values, such as 'promotion of education for all' and "sowing the 
seeds for the future” by educating everyone to become good members of society.' 

The document also states that 'A lack of good communication could lead to perceptions that 
values such as justice, equality and trust are not being upheld' 

The following extract summarises the key values to be considered: 


9 

Key societal values to be considered across public 
health, social care and health care: 

• Accountability 

• Collective responsibility 

• Dignity 

• Education 

• Fairness 

• Honesty 

• Humanity 

• Individual rights 

• Justice 

• Maximising total benefit/benefit for most/utilitarianism 

• Quality of life 

• Respect 

• Right to health and welfare for all 

• Safeguarding the vulnerable 

• Value/quality of service 


72 http://web.archive.ora/web/201 51 1 20091 727/http://www. nice. orq.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-equitv-efficiencv.pdf 

73 http://web.archive.ora/web/2Q1 51 1 20091 727/http://www. nice. orq.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-equitv-efficiencv.pdf 
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This extract contains the prioritised values from public health discussions: 


Prioritised values from our public health discussions: 

• Accountability 

• Collective responsibility 

• Education 

• Individual rights 

• Maximising total benefit/benefit for most/utilitarianism 

• Safeguarding the vulnerable 

• Consequentialism 

• Freedom/liberty 

• Respect for the majority/democracy 


1/1/ays of getting involved 

NICE clearly presents the ways that the public can be involved, including some of the ways 
outlined above 74 : 

Get Involved 


We want you to be involved in our work. Tell us what matters to you, your organisation or your community. 


Register as a stakeholder 

Stakeholders are organisations who register with 
NICE to help develop our guidelines or quality 
standards. Read our step-by-step guide to get 
started. 


Get started 


Come to a meeting 

Advisory committee meetings, technology appraisal 
appeal hearings, public board meetings and some of 
our other meetings are open to the public and press. 
Come along and see how we work. 

Attend a meeting 


Tenders 

Register on the Contracts Finder website to receive 
email alerts for open tenders. 

Find out more 


Comment on a consultation 

All guidelines and quality standards are open for 
consultation during their development. Registered 
stakeholders can comment on our recommendations. 


View consultations 


Citizens Council 

Made up of members of the public, our panel debates 
the overarching moral and ethical issues around our 
work. 

Read our reports 


Talk to us 

Have a question or comment? Contact us by 
telephone or email, or follow us on Twitter 
@NICEcomms. 

Get in touch 


Join a committee 

Our guidance and quality standards are all developed 
by independent committees. Join and have an active 
role in producing a guideline or quality standard. 


Take part 


Be an ambassador 

Join our growing network of professionals working in 
health, public health and social care. Help us to share 
our work. 

Fellows and scholars 
Student champions 

Work with us 

We have lots of opportunities to join our friendly and 
busy teams. Take a look at our current vacancies and 
see how you can contribute to shaping the future of 
health and social care. 

View current jobs 


A more complex explanation of the appraisal process (including where the public can be 
involved) is given for both the Single Technology Assessment (ST A) and the Multiple 
Technology Assessment (MTA). 


74 http://web.archive.org/web/2Q1 51 1 201 1 4236/http://www. nice. ora.uk/aet-involved 
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Here is an extract from the STA process (the full processes can be found on the NICE 
website) 75 : 



NICE in Australia 

The Government of Western Australia achieved accreditation from NICE International for the 
process used to produce its Diagnostic Imaging Pathways (DIP) in April 201 2 76 


75 http://web.archive.org/web/2Q1 51 1 20063506/https://www. nice. org.uk/article/pma1 9/chapter/3- 
The-appraisal-process#sta-initiation-and-evidence-submission 


Key elements 


• Appraisal committee meetings are open to the public, which promotes public 
understanding of how evidence is assessed. 

• The Public Involvement Programme (PIP) assigns a Public Involvement Adviser to 
each appraisal and supports patient and carer consultee organisations, their 
representatives, and individual patients or carers throughout the appraisal. 

• When STA are taking place, Appraisal Consultation Document(s) (ACD) are published 
on the NICE website to allow access to the public, (including experts from different 
fields), to submit their comments. A lay version of the ACD for the public, known as 
'Information for the public 1 ) is published on its website. This inclusive process allows 
for different perspectives to be denoted and published for the Advisory Committee to 
take into account. 

• The public can appeal recommendations 

• The Citizens Council 'is one of the few examples of broader public engagement (as 
opposed to specific patients or patient advocates)' (Messina, A022) 77 . 

• 

Limitations 

Despite a considerable amount of information being placed online during the appraisal 
process, there are no online learning, training or development resources for public audience, in 
particular regarding how to assess evidence, and how to produce a proper written submission 
or proposal. 


76 http://web.archive.org/web/2Q1 51 1 2006421 2/https://www. nice. ora. uk/about/what-we- 
do/accreditation/case-studies/aovernment-of-western-australia 

77 https://drive.aooale.com/open?id=OBwQTUaazNAv5WEZNN21vejlOQkU 
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Other significant models 

Canadian Agency for Drugs and Technologies in Health (CADTH ) 

The Canadian Agency for Drugs and Technologies in Health states that: 

'Listening to patients is a vital part of CADTH's work. We recognize that patients have 
unigue knowledge about what it is like to live with a specific disease or medical 
condition. They can describe advantages and disadvantages of current therapies, 
which may not be reported in published literature. And they can tell us what they would 
most value from a new treatment' 78 . 

They also recognise 'stakeholders' as 'individuals and organizations — with important insights 
to contribute to our work'. In order to 'capture all of this evidence' they have two ways of 
involving the public: 

• Input is solicited at the outset of a project and is usually reserved for patient groups. 

• Feedback is reguested after an initial recommendation or draft report has been 
produced. Eligibility to provide feedback varies depending on the CADTH product, but it 
is often open to anyone with an interest in the work. 

Access to outpatient prescription drugs is 'secured through public and/or private insurance 
plans, with the federal, provincial, and territorial governments offering varying levels of 
coverage, with different eligibility reguirements, premiums, and deductibles' (Messina, A022) 79 . 
The Common Drug Review (CDR) 'makes recommendations for formulary listing of new drugs 
to the 1 8 participating publicly funded drug plans', with recommendations 'based on the 
effectiveness, safety, and cost effectiveness of the medication under review compared with 
existing therapies (including non-drug therapies)' (Messina, A022) 80 . 


78 http://web.archive.org/web/201 51 1 24003854/https://www.cadth.ca/provide-input 

79 https://drive.qooqle.com/open7icHOBwQTUqazNAv5WEZNN2lveilOQkU 

80 https://drive.qooqle.com/open?id=OBwQTUqazNAv5WEZNN2lveilOQkU 


Below is an extract from the website which summarises ways of getting involved 81 : 


CADTH Common Drug Review Patient Group 
Input 

The CADTH Common Drug Review (CDR) undertakes rigorous, objective reviews of drugs 
and provides formulary listing recommendations to Canada’s publicly funded health plans 
(except Quebec). 



CADTH Pan-Canadian Oncology Drug Review 
Patient Group Input and Feedback 

The CADTH pan-Canadian Oncology Drug Review fpCODR) undertakes rigorous, objective 
reviews of cancer drugs and makes funding recommendations to Canada's provinces and 
territories (except Quebec). 



Other Stakeholder Feedback 

CADTH produces a variety of different types of reports related to health technologies, 
including prescription drugs and medical devices. CADTH frequently invites feedback on 
this work from interested parties. 


81 http://web.archive.org/web/201 51 1 24003854/https://www.cadth.ca/provide-input 
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Scottish Medicines Consortium 


The Scottish Medicines Consortium (SMC) has a webpage titled 'public involvement', but 
doesn't use the term public on the page, with the heading 'Capturing the patient and carer 

• oo. 

voice 


Understanding the experiences of patients, their families and carers is a key element in 
our decision making process. It is important for SMC members to fully understand 
how a new medicine impacts the guality of life of patients and carers. This enables 
them to make a fully informed decision on whether or not to recommend a new 
medicine. Patients, members of their families and carers can provide unique knowledge 
about what it's like to live with a condition. They can explain advantages and 
disadvantages of medicines that may not be available in the published literature or 
quality of life measures. We work in partnership with patient groups to gather this 
information through our patient group submissions. 


Below is an extract from their webpage which summarises ways of getting 


involved 83 : 


Patient Group Partners 

How we work in partnership 
with patient groups to gather 
the views and experiences of 
patients and carers 


Forthcoming Submissions 

This section contains 
submission that have been 
scheduled for assessment over 
the coming 2/3 months 


Submission form and 
guidance 

Information to help guide 
patient groups through the 
submission process 


SMC explained 

Summary of the role of the 
SMC 


The SMC also has a ‘Public Involvement Network' (PIN) which is 'made up of patient and carer 
groups who use their knowledge, experience and direct contacts with patients and carers to 
ensure the views of patients, carers and members of the public are captured and used to 
inform SMC processes'. SMC states that: 

The PIN Advisory Group helps SMC to continuously improve how we involve patients, 
carers and members of the public in our work. Membership includes four PIN patient 


82 

http://web.archive.org/web/201 51 1 2400491 8/https://www.scottishmedicines. org.uk/Public_lnvolvem 
ent/PublicJnvolvement 

83 

http://web.archive.org/web/201 51 1 2400491 8/https://www.scottishmedicines. org.uk/Public_lnvolvem 
ent/PublicJnvolvement 


group partners, each of whom has active contact with patients and carers and 
previous experience of submitting to SMC. To make sure we have a wide range of 
representation in the group, one representative is nominated through each of the 
following umbrella organisations: Scottish Cancer Coalition, Alliance, Genetic Alliance 
UK and Carers' T rust. The group also includes all SMC Public Partners and some 
members of the SMC team, including an Area Drug and Therapeutics Committee 
(ADTC) representative and an SMC committee member. 

According to the terms of reference, the purpose of the advisory group is to 'strengthen future 
public involvement and encourage patient groups to participate in' the work of the SMC 84 

The SMC also has 'three public partners who are volunteer members of the public working as 
part of SMC Public Involvement team ' 85 They are also full voting members of the SMC 
committee and their role is to 'help ensure that the views of patients, carers and members of 
the public are taken into account during all SMC decision making processes'. 

The public partner has a variety of responsibilities which include: 

• Preparing a presentation of patient group submissions which accurately highlights key 
issues and presenting these submissions during monthly SMC committee meetings. 

• Reading SMC committee papers in advance of attending monthly SMC meetings in 
order to be able to make an informed decision as a voting member of SMC. 

• Participating in Patient and Clinician Engagement (PACE) meetings to help ensure that 
patient and carer perspectives are fully explored and discussed during the PACE 
meeting and that the views of the expert advisors in attendance are appropriately 
reflected in the resulting PACE documentation. 

SMC committee meetings are 'open for members of the public to observe ' 86 


84 

http://web.archive.org/web/201 51 1 2400591 6/https://www. scottishmedicines.ora.uk/Public Involvem 
ent/PIN Advisory Group Terms of Reference.pdf 

85 

http://web.archive.ora/web/201 51 1 2401 0226/http://www.scottishmedicines. ora.uk/Public Involveme 
nt/SMC explained/Public involvement within SMC 

86 

http://web.archive.org/web/201 51 1 2401 0226/http://www.scottishmedicines. ora.uk/Public Involveme 
nt/SMC explained/Public involvement within SMC 
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Relevant generic models of involvement 


This section contains a summary of models of public involvement and engagement which, 
while relevant to health technology assessment, are more generic. 

Please note, when referring to models which use the word 'consumer', this section will use 
that language to reflect the models being described. 

Understanding whose voices need to be heard 

It is important to acknowledge that a number of voices need to be heard to ensure that the 
perspectives of as many people as possible are incorporated in an organisation seeking the 
views of the public. 

Similarly, it is important to acknowledge the 'screaming silences' from people who may be 
unintentionally marginalised 87 . 

Several studies have explored this idea. For example, a consumer participation resource guide 
for organisations, produced by the Department of Public Health and Flinders University in the 
early 2000s, proposed a continuum of voices from the individual to the community voice 88 . 
See Figure 1 below. 

Figure 1: Voices 


Individual 


Community 


Individuals who are receiving or have received health care services 

Carers and/or family members who support individuals who receive health care 

Groups of consumers (who may share a common experience or chronic illness) 

Consumer organisations including advocacy, self-help and consumer network 
organisations 

Potential consumers such as those with unmet needs or from population groups 
with particular needs or access issues 

Members of the community including future users and the wider community that 
benefits from health care services 

Taxpayers and citizens who ultimately pay for services 


87 

http://web.archive.org/web/2Q1 51 1 26060635/http://irn. sagepub.com/content/earlv/201 0/1 1/11/1 744 

987110387741 

88 

http://web.archive.org/web/201 51 1 20041 252/http://healthissuescentre. org.au/images/uploads/resou 
rces/lmproving-health-services-through-cp-resources-guide-for-orgs.pdf 


The guide explains that: 

As you move down the list, the focus spreads outward from the individual and comes 
to effectively embrace the whole community. For this reason consumer participation 
at these broader levels is also referred to as community participation. All these voices 
have a potential contribution to make through involvement in your health care 
organisation. Consumers are diverse, but be aware that participation approaches can 
sometimes be limited to ways that appeal to certain ethnic and socio-economic 
groups only. (Department of Public Health, Flinders University, and the South 
Australian Community Health Research Unit, p.4) 

The guide also includes The ladder of participation', which is a useful way of locating the 
degree of control the public have over a process, for example, the MBS Review: 


Degree 

Participants' action 

Illustrative mode 

High 

i 

i 

Have control 

Organisation asks community to identify the problem and to 
make all the key decisions on goals and means. Willing to help 
community at each step to accomplish goals. 



Have delegated 

Organisation identifies and presents a problem to the community, 
defines the limits and asks community to make a series of 
decisions, which can be embodied in a plan it can accept. 



Plan jointly 

Organisation presents tentative plan subject to change and open 
to change from those affected. Expects to change plan at least 
slightly and perhaps more subsequently. 



Advise 

Organisation presents a plan and invites questions. Prepared to 
modify plan only if absolutely necessary. 

Low 1 

' 

Are consulted 

Organisation tries to promote a plan. Seeks to develop support 
to facilitate acceptance or give sufficient sanction to plan so 
administrative compliance can be expected. 


Receive information 

Organisation makes a plan and announces it. Community is 
convened for informational purposes. Compliance is expected. 


None 

Community not involved. 


Cancer Australia, in their National Framework for Consumer Involvement in Cancer Control 
(2012: 11-12), emphasise the need to identify whose voices need to be heard., Cancer 
Australia developed a typology of the consumer voice and the roles consumers assume when 
engaging with a health service or organisation. For example, consumers may be asked to 
participate in a focus group or tell their story, a role relating to their personal engagement in 
treatment or managing their illness and what the organisation can learn from this. 

On other occasions, consumers may engage as advocates on behalf of a range of people 
affected by cancer. Or they may be asked for their views as advisors or experts due to their 
level of knowledge of issues concerning people affected by cancer. Alternatively, they may be 
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asked as partners based also on their key knowledge of the health system as an end-user. 
See Figure 2 below. 


Figure 2: Typology on the consumer voice, Cancer 
Australia 


Refers to consumers/organisations 


valued for significant 
knowledge as end-user 


Expert 


► 


being recognised for high 
quality level expertise 


Advisor 


► 


giving experiential advice 
to influence decisions 


Advocate 


representing the broad views and experiences 
of a range of people affected by cancer 


Personal engagement 


providing a personal perspective through story- 
telling, surveys, focus groups and targeted groups 


Another model is from the Health Research Authority in the UK; it has published a statement 
about public involvement in research 89 . This may be useful to understand whose voices need 
to be heard and in which circumstances. For example, some researchers, health services or 
organisations would benefit from involving the public in general - precisely because they are 
not affected by a specific health condition. On other occasions, they may require the 
involvement of people affected by a specific condition, those who provide unpaid care or have 
an experience or relationship with a person affected by a health condition. Finally, they may 
benefit from partnerships with health condition-specific consumer groups, community groups 
or charities. 


http://web.archive.org/web/201 51 120041 757/http://www. hra.nhs.uk/documents/2013/10/hra- 
public-involvement-strateqy-circulation-september-2013.pdf 


Here is an extract from the Health Research Authority Statement: 


Appendix A: What do we mean by public involvement in health research? 

A1 By public involvement we mean a range of activities that enable patients 
and the public to have a say in decisions about the way health research is planned, 
designed, delivered, developed, evaluated, managed and regulated. It also means 
where patients and the public are actively involved in the conduct of research 
studies. 

A2 Public involvement means that work is undertaken ‘with’ or ‘by’ patients and 
the public rather than ‘to’, ‘about’ or ‘for’ them. 

A3 By public involvement we do not mean participation, where people are 
recruited to and take part in research studies as the subjects of the research, and 
contribute the information or data used to answer the question being addressed. 

A4 We recognise that other people and organisations use a range of terms for 
what we refer to as public involvement in their own activities, including public 
engagement and public participation. However, we have decided to use the term 
public involvement for this to be consistent with the Department of Health, National 
Institute for Health Research (NIHR) and majority of organisations who fund and 
manage health research in the UK. 

A5 We will use the term public involvement as a short-hand term for involving 
patients and the public in health research. When we use this term public means 
patients, potential patients or members of the public including those with known 
genetic dispositions, carers and people who use health and social care services as 
well as people from organisations that represent people who use health and social 
care services. 

A6 Where necessary we will define what we mean by public depending on who 
is taking part in the involvement activity. For instance, there may be times when 
patients and the public are involved, or where what is needed is the general public 
and not patients, or other times where patients and carers are needed and not the 
general public. 

A7 We will not use any abbreviations for public involvement, such as PPI (for 
Patient and Public Involvement). This is to ensure that it is always clear that we 
mean involvement and that it is about patients and the public. It will also help avoid 
developing a jargon term that is misused, for example “PPI people”, “PPI 
involvement”, “PPI-ing”. 

A8 The way in which patients and the public can be involved in the work of 
organisations varies. Different methods need to be used appropriate to the 
circumstances and audiences, which can vary in depth, influence, and timescale. 
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National and international frameworks 


There are several national and international frameworks for 'Patient Centred Care', 'public 
involvement', 'citizen involvement' and 'consumer engagement'. 

Institute for Patient and Family- Centered Care (US) 

The best-known practice model in terms of Patient Centred Care (PCC) is from the Institute 
for Patient- and Family- Centered Care (US). 

While this model is targeted at the delivery of care, the principles of involvement (respect, 
information sharing, participation and collaboration) are highly transferable to the HTA 
process, with pertinent points highlighted in bold. 

'Advancing the Practice of Patient- and Family-centred Care in Primary Care and Other 
Ambulatory Settings' is a guide for primary care and other ambulatory settings. The Institute 
outlines a framework, which includes four core concepts of PCC, and a series of 
recommendations for health services hoping to embed PCC in their practice 90 . While the focus 
is on patient-centred care, it has significant relevance to the MBS. Significant parts of this 
model are summarised below. 

The four core concepts or principles of PCC are: 

1. Respect and dignity 

2. Information sharing 

3. Participation 

4. Collaboration 

1. How to embed dignity and respect into practice? 

• clinicians and staff individualise care considering patient and family goals, priorities and 
values 

• care and referrals are coordinated across departments and specialists. 

• staff understand, and position descriptions include, the value of a patient centred 
approach to health care 

• staff with skills and attitudes aligned with patient centred care are employed by the health 
service 

• staff training includes communication skills and patient centred care principles and 
practice 

• patient and family experiences of care are collected systematically and analysed for 
quality improvement (e.g. using storytelling, interviews, recordings, video clips, writing, 
visual arts, etc.) 


90 http://web.archive.org/web/201 51 1 20042447/http://www. ipfcc.org/pdf/GettinaStarted- 
AmbulatorvCare.pdf 


• families are welcome to be with the patient as much as the patient chooses 

• visiting hours (for non-family members) are flexible and take into consideration patient 
and family preferences 

• there are strategies to address the cultural diversity of patients and families (e.g. cultural 
awareness training, getting to know your community workshops, interpreters, etc.) 

• patients and families are involved in the design of facilities to respect diversity, privacy 
and comfort; they feel welcomed into the health service (e.g. clear signage, parking) 


2. How to embed information sharing into practice? 

• patients and families receive timely and clear information about medications, lab reports, 
x-rays and other tests (e.g. using medication lists and hand-held medical and clinical 
records) 

• information is provided in formats, media, language and reading levels appropriate to the 
patients and families 

• patients and families are involved in the review and development of information (written, 
web-based, DVDs, etc.) 

• the health service uses a well-written health information guide when producing 
information for patients and families (e.g. Victorian Department of Health 'Well-written 
health information guide') 


3. How to embed participation in practice? 

• patient and families are seen as essential members of the heath care team 

• family members are encouraged and supported to be involved in care planning, decision- 
making about care and treatments and discharge processes 

• patients and families are encouraged to be present and participate in rounds and nurse 
shift changes 


4. How to embed collaboration into practice? 

• clinicians, staff and leadership of the organisation support a patient centred, collaborative 
approach and welcome working in collaboration with patients and families in: 

governance (e.g. Board, executive team) 

committees and teams (e.g. safety and quality, ethics, research, buildings and 
grounds, catering) 

professional development (e.g. training for staff) 
staff selection and performance appraisals 
review of complaints and compliments processes 
review of health information 

• patients and families involved in collaborative initiatives (committees and teams) receive 
support and training (e.g. reimbursement for out-of-pocket expenses, timely agendas and 
minutes, briefing and debriefing, orientation, skills development opportunities, 
acknowledgement of contributions, etc.) 

• patients and families establish an advisory group for the health service and receive 
support from the health service (e.g. administrative, secretariat, reimbursement for out- 
of-pocket expenses, peer support, mentoring, etc.) 
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• the health service has systems and processes in place to support collaboration (e.g. 
policy, staff employed to lead and support the collaboration, annual budget allocation, 
evaluation). (Institute for Patient- and Family- Centered Care, 2011) 

The ladder of citizen participation' 

The 'ladder of citizen participation' was developed in the late 1960s by Sherri Arnstein 91 . She 
envisaged eight levels of citizen participation. Please note, this report has produced an 
updated, action-based model in the section 'Social determinants of involvement'. 

Figure 3: Ladder of citizen participation, Sherri Arnstein 92 







Citizen control 

_ , . - Degrees of 
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George Julian 

V— » Knowledge Transfer Consultancy 


The levels of participation are: 


• Manipulation and Therapy are both non-participative; the aim of these interventions is 
to cure or educate the participants or to achieve public support by public relations. 


• Informing is a most important first step to legitimate participation, but too frequently 
the emphasis is on a one way flow of information, there are no channels for feedback. 


• Consultation is a legitimate step and includes, for example, attitude surveys, 
neighbourhood meetings and public enquiries. 


91 https://www.plannina.ora/pas/memo/2007/mar/pdf/JAPA35No4.pdf 

92 http://web.archive.ora/save/http://www.qeorqeiulian. co.uk/2013/01/22/social-media-and-citizen 
engagement/ 


• Placation, for example, co-opt and hand-pick community members onto committees; it 
allows citizens to advise or plan, but the power-holders retain the right to judge the 
legitimacy or feasibility of the advice. 


• Partnership implies that the power is in fact redistributed through negotiation between 
citizens and power holders; the planning and decision-making responsibilities are 
shared, e.g. through joint committees. 


• Delegated power implies that citizens hold a majority of seats on committees and have 
delegated powers to make decisions. Public now has the power to assure 
accountability of the programs. 


• Citizen Control is when the have-nots handle the entire job of planning, policy making 
and managing a programme, e.g. neighbourhood corporation with no intermediaries 
between it and the source of funds. (Arnstein S 1969) 

The International Association for Public Participation 

The International Association for Public Participation (IAP2) developed a public participation 
spectrum 93 , which reworks the traditional 'ladder of participation'. It is widely used in urban 
planning and by councils and municipalities in planning in collaboration with citizens in their 
catchments. Although not specific to the health sector, the spectrum of participation is useful 
for all public services, including health services and organisations in the health sector. It 
divides the 'spectrum of participation' into 'goals' and 'promises' and gives examples of tools 
which can be used to achieve these. 


93 http://web.archive.ora/save/https://www. iap2.ora.au/resources/iap2s-public-participation-spectrum 
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Figure 4: IAP2 public participation spectrum, International Association for Public Participation 


94 



Patient and Family Engagement (PFE) Framework 

The American Institute for Research developed the Patient and Family Engagement (PFE) 
Framework as an instrument to help drive actions toward outcomes 95 . These outcomes are: 
better health outcomes, improved population health, better health care quality, cost-effective 
care, and a more patient centred healthcare system. The PFE Framework shows how the role 
of patients and families can be expanded across all critical areas of health care: at the point of 
care, in health care organisations, and in policy-making. See Figure 5 below. 


94 http://web.archive.ora/save/https://www. iap2.ora.au/resources/iap2s-public-participation-spectrum 

95 https://web.archive.org/web/201 51 126040401 /http://aircpce.org/model-patient-familv- 
enqaqement/ 



Figure 5: PFE Framework, American Institute for Research 
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Factors influencing engagement: 

• Patient (beliefs about patient role, health literacy, education) 

• Organization (policies and practices, culture) 

• Society (social norms, regulations, policy) 


96 https://web.archive.ora/web/201 51 1 26040401 /http://aircpce.ora/model-patient-familv- 
enaaaement/ 
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The Engagement Cycle 

InHealth Associates is a UK network of patient centered improvement specialists and 
experienced patient and public engagement practitioners. They developed the 'Engagement 
Cycle' as a useful model to use for planning and developing an engagement and involvement 
strategy 97 . 



In Health 

ASSOCIATES 


There are ten key elements of the 'Engagement cycle': 


97 https://web.archive.org/web/201 501 1 4024258/http://enqaqementcvcle.ora/enqaqement-strateqv- 
culture-and-svstems/ 




1 . Be clear on the vision and build shared understanding 

The model can contribute towards a shared understanding of what engagement means, 
its different purposes and benefits. It helps develop the principles underpinning a vision, 
strategy and plans for local engagement. 

2. Develop an engagement strategy that links to other work 

The Engagement Cycle can be used as the basis for developing a strategy on engagement 
and should be linked to other organisational plans. 

3. Be clear about who needs to do what 

Roles and responsibilities for engagement at strategic and operational level need to be 
clear. This includes management, Boards, lay members, team leaders, and community 
members. 

4. Embed engagement in governance arrangements 

The Board requires processes to ensure that information gained from engagement is used 
to provide insight (into current services) and foresight (for strategic planning). 

5. Ongoing mentoring and evaluation 

An assessment of engagement activities (outcomes, processes, mechanisms) should be 
undertaken for each stage of the cycle. Involve patients, carers and the public in gathering 
data, analysing it, and in decisions about what happens next. 

6. Develop meaningful engagement structures 

Think 'function before form' and build on what already exists. Any structure (e.g. consumer 
advisory group) should have clear purposes. Use consultation and engagement 
mechanisms such as those described in the Participation compass. 

7. Build trusting relationships 

At each stage of The Engagement Cycle, it is crucial to develop dialogue with local patient 
and community organisations and patient leaders. Take every opportunity to build links 
with local people and groups - be ready to 'walk the talk' and be able to explain clearly 
what you are doing. 

8. Provide adequate resources 

• To move engagement from a 'nice to have' to a 'must do' means providing sufficient 
resources for engagement within operational programs and team budgets. This includes 
payment, reimbursements, acknowledgements and incentives for patients and the public, 
as well as investment in training and support. 

9. Provide learning and support to staff 

Different people require learning and support in order to implement The Engagement 
Cycle. It is important to think through who needs learning and support in order to develop 
confidence, and skills in this area, for example Board members, team leaders, health 
professionals and clinical staff. 

1 0. Provide learning and support to patients, carers and the public 

Providing learning and support for patients and the public is crucial. There are many sorts 
of patient leaders who also need opportunities to build their capacity to engage in 
dialogue, be effective and build trusting relationships. 
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Framework for Consumer Involvement in Cancer Control 

Cancer Australia developed its Framework for Consumer Involvement in Cancer Control in 

201 1 which includes four main key 

elements 98 . 


Committed 

organisations 

Governance, cultural engagement, clear 
objectives, resources and support, 
education and training, facilitation, 
research, data and evaluation 


Shared 

focus 

Accessible information 
and support, delivery of 
quality services and 
best practice, research 
and evaluation, policy 
development, 
benchmarking and 
linkages 


Improving 
outcomes and 
experiences of 
people affected 
by cancer 


Capable 

consumers 

Experience of disease, 
motivation, capability 
to meet environment, 
understanding of 
context, learning and 
development, support 
and networks, 
mentoring 


Inclusive 

groups 


Champions, mutual respect, equality, 
combined learning and development. 


communication 


Below is a summary of each group identified by the model. 

Committed organisations: Committed organisations integrate consumer involvement 
throughout every level of their organisation, service or group. Organisations which are 
committed to involving consumers demonstrate the following: 


98 


http://web.archive.org/web/201 51 1 26041 346/https://canceraustralia. gov.au/sites/default/files/public 
ations/national_consumer_frannework_web_504af020f21 84.pdf 


• governance: a governance structure which incorporates principles, values and 
elements of consumer participation 

• cultural engagement: active involvement of people with respect to their cultural needs, 
and targeted approaches aimed at those culturally diverse communities whose 
outcomes are poorer 

• clear objectives: consumer involvement which has clear objectives in Terms of 
Reference, policies and procedures 

• resources and support: consumers are explicitly resourced and supported to be 
effectively engaged 

• education and training: consumers are provided with appropriate training and 
development opportunities; staff are trained to strengthen their understanding of the 
benefits of consumer involvement 

• facilitation: active facilitation and coordination of consumer involvement activities, 
including consumer feedback and participation in organisational strategy 

• research, data and evaluation: consumer involvement activities are monitored; 
involvement strategies are researched and evaluated for quality improvement and 
benchmarking. 

Capable consumers: Capable consumers have developed knowledge from their experience 
and are able to represent the views of others. The key characteristics of a capable consumer 
are: 


• consumer experience: consumers have an experience of cancer either as a patient, 
carer, family member or survivor 

• consumer motivation: consumers are motivated to participate in a largely voluntary 
role to improve outcomes for others 

• consumer capability to meet the environment: consumers develop their skills and 
capabilities to meet the requirements of the role 

• consumer understanding of context: consumers develop an understanding of the 
health context where they are involved, and the types of consumer involvement 

• learning and development: consumers undertake learning and development 
opportunities, including conference and forum attendance and co-authoring journal 
articles, to build their expertise in consumer participation 

• consumer support and networks: consumers seek support through connections with 
consumer organisations, networks and support groups 

• consumer mentoring: consumers encourage and support other consumers to actively 
participate and develop in the role of consumer representative 


Inclusive groups: to gain the most out of consumer participation it is important to cultivate an 
environment where consumers feel valued and equal members with others in the group. This 
environment can best be identified as an 'inclusive group'. Inclusive groups which are 
committed to involving consumers demonstrate the following: 

• champions: leaders exist within the organisation who promote the benefits of 
consumer involvement 

• mutual respect: consumers are respected and valued for their contribution. Their views 
are actively sought, listened to and considered. 

• equality: consumers are considered equal members of the team 
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• combined professional development: professionals and consumers undertake 
professional development opportunities together and learn from each other's 
knowledge and experience 

• communication: professionals and consumers communicate in a way that builds 
knowledge, understanding and mutual respect. 


Shared focus: This element means that to have effective consumer involvement in cancer 
control, consumers are active participants, actively supported by organisational systems and 
processes, working as members of a team to achieve objectives. This element draws all 
elements together to focus on outcomes. In order to have a shared focus, consumers may be 
actively involved in the following areas: 

• development of accessible information and support: working together, consumers and 
professionals develop accessible information and support for people affected by 
cancer 

• delivery of quality services and best practice: consumers join with health professionals 
to ensure the delivery of safe and high quality cancer services based on best practice 

• research and evaluation: consumers are involved in the design, conduct, translation 
and evaluation of research 

• policy development: consumers participate in policy development to improve cancer 
outcomes 

• benchmarking and linkages: consumers and professionals develop performance 
measures to help determine the effectiveness of consumer involvement and facilitate 
benchmarking opportunities for quality improvement. 


The framework also provides this model of 'types of consumer involvement'. The framework 
additionally uses a similar 'levels of participation model', comparable to Arnstein, but 
examines organisational capacity and 'consumer capability' 


Advisor 
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Health Canada 


Health Canada developed a model for public involvement based on a continuum of five levels 
of public involvement; from low level involvement to high level involvement": 


Health Canada's 
Public Involvement Continuum 


Level 1 

Level 2 

Level 3 

Level 4 

Level 5 

Low level of 


Mid level of 


High level of 

public Involvement 


public involvement 


* public involvement 
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Gather Information 
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The model outlines the following levels 
Level 1 Inform/educate when: 

• factual information is needed to describe a policy, program or process 

• a decision has already been made (no decision is required) 

• the public needs to know the results of a process 

• there is no opportunity to influence the final outcome 

• there is need for acceptance of a proposal or decision before a decision may be made 

• an emergency or crisis requires immediate action 

• information is necessary to abate concerns or prepare for involvement 

• the issue is relatively simple 

Level 2 Gather information/views when: 

• the purpose is primarily to listen and gather information 

• policy decisions are still being shaped and discretion is required 

• there may not be a firm commitment to do anything with the views collected (we 
advise participants from the outset of this intention to manage expectations) 


99 http://www.hc-sc.qc.ca/ahc-asc/alt_formats/pacrb-dqapcr/pdf/public-consult/2000decision 
eng. pdf 


Level 3 Discuss or involve when: 

• we need two-way information exchange 

• individuals and groups have an interest in the issue and will likely be affected by the 
outcome 

• there is an opportunity to influence the final outcome 

• we wish to encourage discussion among and with stakeholders 

• input may shape policy directions/program delivery 

Level 4 Engage when: 

• we need citizens to talk to each other regarding complex, value-laden issues 

• there is a capacity for citizens to shape policies and decisions that affect them 

• there is opportunity for shared agenda setting and open time frames for deliberation 
on issues 

• options generated together will be respected 
Level 5 Partner when: 

• we want to empower citizens and groups to manage the process 

• citizens and groups have accepted the challenge of developing solutions themselves 

• we are ready to assume the role of enabler 

• there is an agreement to implement solutions generated by citizens and groups 


To summarize, the decision to inform, consult or engage and the related selection of a public 
involvement strategy is dependent on a number of complex, interrelated factors: 

• tailoring of approaches for involvement with goal and phase of policy making 

• level of influence and involvement participants expect to have 

• nature and complexity of issues 

• participant profiles (e.g. mix of citizen vs. group representatives) 

• previous experience of organizers with various techniques 

• level of concern and media attention around the issues 

• timelines 

• financial costs 

• human Resources and expertise 

• degree of federal/provincial/territorial collaboration required 

• level of support from stakeholders/partners 

• level of political support in department or across government. (Health Canada 2000, 
p.12-14) 
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Health Consumers NSW 


In 201 5, Wentwest and Health Consumers NSW developed a Consumer and Community 
Engagement Model to identify how to improve consumer-centred care in the primary care 100 . 
The model incorporates a 'top down' and 'ground up' approach. The model outlines how 
consumers are involved in governance; describes the guiding principles and values of public 
involvement; and the critical inputs and key outcomes of the engagement process. 
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100 

http://web.archive.org/web/201 51 1 20051 1 52/http://www. hcnsw.org.au/data/Resources/2Q1 5_07_1 7. 
Final_report_and_template.pdf 


Health Issues Centre 


The Health Issues Centre developed a model for consumer engagement for the Southern 
Melbourne Integrated Cancer Services (Victoria) 101 . This model was for consumer 
engagement in guality improvement. 

The 'pin-board' represents the policies and frameworks that inform the planning and 
implementation of consumer participation. 

The 'pot plant and watering can' represent the main principles that are reguired to guide 
planning and implementation. 

The 'filing cabinet' consists of three drawers titled partnership, consultation and information. 
These drawers signify three approaches to consumer participation and their contents include 
a variety of methods that can be used for participation. 

The 'easel' suggests that the whole process of planning and implementing consumer 
participation needs to include an ongoing evaluating and monitoring process. 


101 

http://web.archive.org/web/201 51 126042923/http://www. healthissuescentre.ora.au/imaaes/uploads/ 
resources/Consumer-participation-resource-for-researchers.pdf 
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Significant themes 


This section identifies significant themes which were identified during the literature review. 

The themes identified were: 

• Social and ethical issues 

• Public Involvement 

• Learning, training, education and development 

Summary of documents relevant to significant themes 

While many documents agree and overlap, the following table summarises which documents in particular explored significant themes identified 
in the review. 


Theme 

REF 

Link 

Issuing Entity 

Title 

Social and 
ethical issues 

A01 8 

httDs://drive.aooa 

le.com/oDen?id=0 

BwOTUaazNAv5R 

kxwVXVZRIRKSU 

E 

Social Sciences & 
Medicine 

Eliciting ethical and social values in health technology assessment: A participatory 
approach 


A024 

httDs://drive.aooa 

le.com/oDen?id=0 

RwOTUnazNAvSY 

WRiZ2JMNDZBR 

IA 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Socio-ethical issues in personalized medicine: a systematic review of English 
language health technology assessments of gene expression profiling tests for 
breast cancer prognosis 

Policy, decision 

A001 

httDs://drive.aooa 

le.com/c>Den?id=0 

Health Policy 82 

Bringing 'the public' into health technology assessment and coverage policy 
decisions: From principles to practice 
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making 
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A002 

http://web. archive 
.ora/web/201 51 1 
06063004/httD:// 
www.biomedcent 
ral. com/content/o 
df/sl 2911-01 5- 
01 76-0. Ddf 

BMC Medical 
Informatics and 
Decision Making 

Public preferences for engagement in Health Technology Assessment decision- 
making: protocol of a mixed methods study 

Public 

involvement 

A005 

httDs://drive.aooa 
le.com/open?id=0 
BwQTUaazNAv5Z 
TB1 ZzVCcm9Cb2 
s 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Perspectives on health technology assessment: response from the patient's 
perspective 


A007 

httDs://drive.aooa 

le.com/open?id=0 

BwOTUaazNAv5V 

2FwbDh6MTIsZG 

c 

Health Policy 69 

A consumer involvement model for health technology assessment in Canada 


A01 0 

httDs://drive.aooa 

le.com/oeen?id=0 

BwOTUaazNAv5c 

0R6a3l1Y19acrmc 

BMC Health Service 
Research 

Study Protocol: Introducing patient perspective in health technology assessment 
at 

the local level 


A01 2 

httDs://drive.aooa 

le.com/oeen?id=0 

BwOTUaazNAv5L 

UMxM09nVXRiOk 

U 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Survey on the involvement of consumers in health technology assessment 
programs 


A01 4 

httDs://drive.aooa 

BMC Health Service 

Involving patients in PITA activities at local level: a study protocol based on the 



le.com/open?id=0 

BwQTUaazNAv5e 

nJkSIIGN2ZrRVU 

Research 

collaboration between researchers and knowledge users 

A01 5 

https://drive.aooa 
le.com/open?id=0 
BwQTUaazNAv5b 
3hu02JWS1 dwT 
m8 

BMC Health Service 
Research 

Involving patient in the early stages of health technology assessment (HTA): a 
study protocol 

A01 6 

https://drive.aooa 

le.com/open?id=0 

BwOTUaazNAv5V 

XhDTUtOZE8zcW 

M 

Social Sciences & 
Medicine 

"It all depends": Conceptualizing public involvement in the context of health 
technology assessment agencies 

A017 

httos://drive.aooa 
le.com/ooen?id=0 
BwOTUaazNAv5c 
1 RVLVVERiZnbGs 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Consumer involvement in the health technology assessment program 

A022 

httos://drive.aooa 

le.com/ooen?id=0 

BwOTUaazNAv5 

WFZNN2lveil00k 

U 

Patient 

A Pilot Study to Identify Areas for Further Improvements in Patient and Public 
Involvement in Health Technology Assessments for Medicines 

A025 

httos://drive.aooa 

le.com/ooen?id=0 

BwOTUaazNAv5Z 

klvWiFKdlnNdWM 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Health technology assessment use and dissemination by patient and consumer 
groups: Why and how? 

A028 

httos://drive.aooa 

le.com/ooen?id=0 

International 
Journal of 

Does the public think it is reasonable to wait for more evidence before funding 
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Key learning points from significant themes 


This section attempts to articulate key learning points from the literature review, in order to 
help inform decision making about best-next steps. It is organised according to the significant 
themes explored in the previous section. 

Social and ethical issues 

The public interest 

Natural balances need to exist between Government, acting in the public interest and 
'politically accountable for their resource allocation decisions' (Abelson, AOOI) 102 , and industry, 
often acting in the interest of private investors. Public involvement 'increases transparency' 
(Wortley, A002) 103 . 

A key question that arose in the review was regarding pricing. Aside from 'differential pricing 
between industrialised and developing countries', members of the public ask if drugs are 
priced at what they cost to produce, or what the market is willing to pay (T rouiller, A050) 104 . 

'Engagement with an industry whose strategy has so far largely been to maximise 
profit in the West, rather than establish an equitable pricing policy worldwide, requires 
careful management of intellectual property' 

The ethical and moral debates around this rely on transparency on both sides. An organisation 
attempting to bridge this gap is the European Patients' Academy, which aims to: 

'help patients be more educated and involved in the research and development 
process of new medicines by offering reliable, objective, comprehensive lay-friendly 
information and training on the research and development process of medicines. We 
will increase the capacity of patients to be effective advocates with meaningful 
involvement in areas like drug discovery and non-clinical testing, planning and conduct 
of clinical trials, regulatory affairs, assessment of safety of medicines, benefit-risk 
assessment, as well as principles of health technology assessment.' 105 


102 https://drive.aooale.eom/file/d/0BvvGFVQF01 AMNXZQMXZDQkhIMI U/view 

103 

http://web.archive.org/web/201 51 1 06063004/http://www. biomedcentral.com/content/pdf/sl 2911- 
01 5-01 76-0. pdf 

104 

http://web.archive.org/web/201 51 1 1 71 02634/http://fieldresearch.msf.ora/msf/bitstream/1 01 44/2844 
1 /I /Access%20T rouiller%202002.pdf 

105 http://web.archive.org/save/http://www.patientsacademv.eu/index.php/en/about-eupati/8-patients- 
academv-a-new-wav 


The MBS Review Consultation Paper states that 'the Review is clinician led' in order to 
'minimise real and perceived conflicts of interest' 106 , yet lacks any independent review, 
comparable to the NICE Citizens Council. 

The same report makes reference to the 'Choosing Wisely' model, which is: 

'founded on the principles of managing conflicts of interest, improving the quality of 
care, improving access to care, and promoting the just distribution of finite resources. 

It brings together clinicians and consumers and considers services which are of 
questionable value, to inform the decisions of clinicians and to empower consumers 
to participate in conversations with their doctor about their care' 107 

The Choosing Wisely campaign will be 'facilitated' and evaluated in Australia by NPS 
MedicineWise 108 (known prior to 2009 as the National Prescribing Service 109 ), who 'work with 
health professionals, consumers, government and industry to improve the health of all 
Australians through the quality use of medicines and medical tests'. It is interesting to note 
that they state 'We also work extensively with consumers,' without giving any detail or clear 
ways for people to get involved. 

The 'bringing together' of the public (or representatives) with subject experts in a transparent 
way is key to maintaining trust in the appraisal process. The report by the NICE Citizens 
Council states that the transparency of such processes can affect 'perceptions that values 
such as justice, equality and trust are not being upheld' 110 . 

Public involvement 

Failure of evidence-based medicine to involve the public 

Fundamental biases in evidence-based medicine (and thus any organisation informed by this 
evidence) are proposed in the paper 'Six 'biases' against patients and carers in evidence-based 
medicine' (Greenhalgh, A048) 111 . The authors state: 


106 

http://web.archive.org/web/2Q1 511201 2431 0/http://www. health, aov.au/internet/main/publishina.nsf/ 
Content/922CB2933B0F1 645CA257EC1 001 D5C1 2/$File/MBS%20Review Consultation%20DaDer FIN 
AL.pdf 

107 

http://web.archive.org/web/201 511201 2431 Q/http://www.health. aov.au/internet/main/publishina.nsf/ 
Content/922CB2933B0F1 645CA257EC1 001 D5C1 2/$File/MBS%20Review Consultation%20DaDer FIN 
AL.pdf 

108 http://web.archive.org/web/201 51 1 2701 1 21 7/http://www.nps.ora.au/media-centre/media- 
releases/repositorv/choosina-wiselv-australia-launchina-in-201 5 

109 https://en.wikipedia.org/wiki/NPS MedicineWise 

1 1 0 http://web.archive.org/web/201 51 1 20091 727/http://www. nice. ora.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-equitv-efficiencv.pdf 

111 http://web.archive.org/web/201 51 1 1 7045807/http://www.biomedcentral.com/content/pdf/s1 291 6- 
015-0437-x.pdf 
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'Evidence-based medicine (EBM) is maturing from its early focus on epidemiology to 
embrace a wider range of disciplines and methodologies. At the heart of EBM is the 
patient, whose informed choices have long been recognised as paramount. However, 
good evidence-based care is more than choices. We discuss six potential 'biases' in 
EBM that may inadvertently devalue the patient and carer agenda. To reduce these 
'biases', EBM should embrace patient involvement in research, make more systematic 
use of individual ('personally significant') evidence, take a more interdisciplinary and 
humanistic view of consultations, address unequal power dynamics in healthcare 
encounters, support patient communities, and address the inverse care law.' 

The six biases they identify are: 

• limited patient input to research design, 

• low status given to experience in the hierarchy of evidence, 

• a tendency to conflate patient-centred consulting with use of decision tools; 

• insufficient attention to power imbalances that suppress the patient's voice, 

• over-emphasis on the clinical consultation, 

• a focus on people who seek and obtain care (rather than the hidden denominator of 
those that do not seek or cannot access care). 

In addition to this, Messina also states that 'the exact point to seek input from patients and 
the public is controversial; however, it suggests that 'those involved within HTAs agree that 
patients/the public are usually involved too late in the process' (Messina, A022) 112 . Messina 
adds: 

Involving patients from the start, in trial design, can greatly impact on the relevance of 
the final product and the associated 'evidence.' 


112 Involving patients from the start, in trial design, can greatly impact on the relevance of the final 
product and the associated 'evidence.'! 



Social determinants of involvement 


While the social determinants of health have been clearly defined by World Health 
Organisation 113 , the social determinants of involvement are equally important to consider 
when involving the public. 

Accepting that you can't involve everyone in everything it is therefore important to consider 
how you can include as many people as possible within your resources. This includes 
considering who you might be unconsciously excluding through certain kinds of involvement 
(for example, face to face meetings might not be possible for some to attend) so it is 
important to consider how you can reduce barriers to involvement. 

The NICE report (explored in the section of this report 'Significant reports by the Citizens 
Council') extensively explores 'tensions between eguity and efficiency ?' 114 

Whatever the 'tension' and 'trade off', it is important that the public are involved in deciding 
where compromises are made and that the decision making process is transparent, so as to 
avoid perceptions of deliberate or unconscious exclusion of certain people. 

It is also important to acknowledge what can be thought of as the 'screaming silences' from 
people who may be unintentionally marginalised 115 . 

The two following resources have been adapted to support members of the Medicare Benefits 
Schedule Review Taskforce appraise their plans for involvement. 

The resource on the next page, adapted from a resource produced by Jack Nunn for 
Macmillan Cancer Support, summarises ways of identifying ways people may be excluded 
from involvement and overcoming certain barriers to involvement 116 

'Supporting public involvement’ was created by Jack Nunn for Macmillan Cancer Support and 
has been adapted for this report. 

What are you doing? Was created by Jack Nunn for the Health Issues Centre and has been 
adapted for this report, informed by the literature review. 


113 http://www.who.int/social determinants/en/ 

114 http://web.archive.org/web/201 51 1 20091 727/http://www.nice.ora.uk/Media/Default/Get- 
involved/Citizens-Council/Reports/cc-reportl 7-equitv-efficiencv.pdf 

115 

http://web.archive.org/web/201 51 1 26060635/http://jrn. saaepub.com/content/earlv/2Q1 0/1 1/11/1 744 
987110387741 

116 http://learnzone.ora.uk/courses/course.php?id=228 
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Supporting public involvement 


There are many things to think about when involving the public and patients in 
improving the Medicare Benefits Schedule - this document is intended to help ask the 
right guestions for the right roles. 


How to use this resource: Under 'Assumptions and barriers', read the questions and 
consider if these might be barriers to involving some people, and consider how you 
might overcome these. 'Learning needs and support' examines the role in more detail 
and asks questions about the support people might need to develop. 


Assumptions and barriers 

• What commitment do you 
expect (time/financial 
implications) 

• Have you asked people to think 
about their emotional readiness? 

• Do you expect them to be 
reading and writing information 
and documents? Have you 
considered what formats might 
be appropriate? 

• Are you assuming a good ability 
to speak and read English? 

• Do you expect a certain 
educational background? 


• Are the people who have 
engaged with you the 
only people who might 
be interested? 


Role Description 

Lay Leader: A person who speaks 
and acts on behalf of all members 
of the public, including patients and 
carers and who takes a leading role 
in representing other lay 
representatives. The role may 
involve holding people or 
organisations to account 

Lay representative: a member of the 
public (not a professional) who is a 
representative. They must speak 
and act on behalf of others. They 
may be guided by lay leaders but 
will be expected to take direct action 
to ensure that they are informed and 
able to represent the views of 
others. 


Interested and engaged members of 
the public: People who know about 
and/or are interested in decisions 
being made, but may take no direct 
action other than giving feedback or 
signing petitions 
Uninformed, disengaged or 
disinterested members of the public: 
people who, for whatever reason, 
are not engaged, informed or 
interested in influencing decision 
making or shaping the future of 
health and social services. 


Learning needs & support 

How are they a representative? 

• How will they be gathering 
views? 

• Will this involve research? 

• Do they have a budget? 

• Should they be paid? 

• Is there admin and practical 
support (from an 
organisation?) 

• Is there any training available? 

Who is already doing this? 

• Are there any opportunities for 
them to be involved in peer 
support or have or be a 
buddy? 

• What can be shared with 
other organisations? (E.g. 
learning, resources) 

How are people involved? 

• Is it face to face meetings? 

• Can people be involved in 
other ways? 

• What can be done online, 
what cannot? 


A majority of the population are in this 
category. 

• What information or support might 
some people need to help them 
move into other roles? 

• What might make people move 
back into this role? (e.g. not 
seeing direct improvements, or 


Could there be a need for translation? 

• Are there any groups or organisations 
who could support with this? 


too much of organisational 
change? 
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What are you doing? 

There are many ways to involve people in the work of the Medicare Benefits Schedule Review. 

How to use this resource: The left hand column shows the kinds of actions you can take, the right hand 
column gives ideas for how people can be involved in these actions. The width of the pyramid is a guide to the 
number of people likely to be involved 


Actions Involve people by: 


Innovating 

This includes prototyping, piloting, 
establishing and creating new ways 
of doing things. This can include 
anything from building partnerships 
or doing research. 


Supporting them to: 

• Design and carry out research 

• Create solutions 

• Implement ideas 

• Learn from actions 


Managing, delivering and 
evaluating 

Working in partnership to manage 
ongoing activities. 



Having: 

• Clear roles for the public 

• Elections and interviews when appropriate 

• Clear and accessible accountabilities for all roles 
(including staff), groups and committees. 

Asking them to take actions such as: 

• Managing or overseeing actions and processes 

• Directly delivering services or reviewing providers 

• Evaluating actions, processes and outcomes 


Prioritising and planning 

Working in partnership to prioritise 
actions and plan implementation. 

Listening, responding 
and acting 

Actively seeking feedback, responding 
to ideas, compliments and complaints 
with actions. 


• Agreeing priorities in a clear, transparent way* 
Proactive . Having clear accountabilities for planning at all stages 

• Having a transparent and adaptable budget 
‘(this can include stopping certain actions) 

Asking them to help: 

• Interpret feedback 

Reactive • Influence responses to ideas, compliments and 
complaints 

• Asking for ideas for actions 


Remember to always take action, however big or small. 
This includes telling people what this action was, particularly 
those who have given feedback. 


Asking and discussing 

Asking people what they think, need 
and want and discussing it with them. 


Inviting people from your intended audience or people you 
are trying to help to: 

• Design how you will collect feedback and interpret 
the results 

• Identify any potential barriers that might stop people 
from giving feedback. 


Telling 

Giving information about what you 
have done, are doing or are going to 
do. 


Sharing opportunities to be involved 

Asking people for ideas and support to share and 

disseminate what you want to tell people 


Before involving people, always ask: 

• Could this be more accessible? (is it online? Does it use appropriate language - tone and tongue?) 

• Is there any way I could support involvement? (learning and development, financial support?) 


Please note the term ‘people’ includes the general public, patients, consumers, carers and service users. 
An ‘action’ is anything people are doing, individually or on behalf of an organisation 

★ 

This is the ‘leading edge'. 



Creating opportunities for learning, education, 
training and development 

Key quotations 

From A Pilot Study to Identify Areas for Further Improvements in Patient and Public 
Involvement in Piealth Technology Assessments for Medicines', Messina 117 

• Education of patient groups and committees was essential in the creation of active 
and informed agents in medicine HTAs. 

• More research, funding, education, and HTA process changes are required to shift 
patients and the public towards the center of the decision-making model. 

• Education and using innovative methods of reaching patients, as well as adequate 
mechanisms for feedback in the process, were noted as suitable options for 
enhancing consultation in HTAs. 

From 'Consumer Participation in the Review of Health Technology Assessment Report 
on outcomes of consumer consultations', Consumer Health Forum 118 

• Consumers suggested that consumer education should be an integral function of the 
new HTA system. 

• Consumer education is insufficient, and that there should also be education of health 

professionals. Health literacy programs could enable them to communicate better 
with health consumers about health technologies. This would improve consumers" 
ability to provide informed consent when their health professionals are proposing the 
use of a technology in their treatment. 


117 https://drive.aooale.com/open7icHOBwQTUaazNAv5WEZNN21veilOQkU 
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Quotations relevant to learning, education, training and development 


REF 

Link 

Issuing 

Entity 

Title 

Quotations relevant to learning, education, training and development 

A004 

httDs://dr 

ive.aooal 

e.com/oD 

en?id=0B 

wOTUaaz 

NAv5NW 

9tZGoYV 

mN3bHM 

European 

Heart 

Journal 

Health technology 
assessment in 
interventional 
electrophysiology and 
device therapy: a position 
paper of the European 
Heart Rhythm 
Associationt 

Issues related to training, education, definition of the appropriate setting for therapy 
delivery need to be considered and addressed. 

A016 

httos://dr 

ive.aooal 

e.com/oo 

en?id=0B 

wOTUaaz 

NAv5VXh 

DTUtOZE 

8zcWM 

Social 

Sciences 

& 

Medicine 

"It all depends”: 
Conceptualizing public 
involvement in the 
context of health 
technology assessment 
agencies 

Public education about HTA "in the interest of allaying some of the anxiety commonly 
expressed by lay people [and to educate] the mass of the people on medical matters, an 
important area of civilised discourse” 

A017 

httos://dr 

ive.aooal 

e.com/oD 

en?id=0B 

wOTUaaz 

NAv5c1 R 

VLVVERi 

ZnbGs 

Internati 

onal 

Journal 

of 

Technolo 

gy 

Assessm 
ent in 
Health 
Care 

Consumer involvement in 
the health technology 
assessment program 

T raining and procedures were developed for staff 'to enable them to take on the role of 
identifying and contacting consumers to comment' 

T raining for fellow panel members, especially chairmen, is valuable. 

A022 

httos://dr 

Patient 

A Pilot Study to Identify 

Involvement of patients and the public is superior to consultative methods, and that 



ive.qoogl 

e. com/op 

en?id=0B 

wQTUgaz 

NAv5WE 

ZNN2lveil 

OQkU 


Areas for Further 
Improvements in Patient 
and Public Involvement in 
Health Technology 
Assessments for 
Medicines 


patients and the public can join the decision- making table. This reguires a nurturing 
climate for participation along with adeguate support during the process and appropriate 
education and training. 

NICE, a recognized leader in patient and public involvement in HTA, has undertaken work 
to evaluate the success of its program, which reveals that it is making a positive impact, 
but training and education were identified as an area for improvement. Education will 
help patient and public advocates develop appropriate skills to participate in HTA debates 
and may also assist in providing professionals with the skills to understand and make 
use of patients' experiences and views. 

Education programs are already underway in Canada and the UK (for example, the 
London School of Economics Summer School) and are under discussion in Australia but 
in- creased funding would ensure these programs are a priority. 

HTA agencies should be responsible for funding public and patient involvement 
initiatives. 

Additional phases of research are also planned in the area of how to better prepare 
patient groups to be part of the HTA process through ongoing training and development. 
Education and using innovative methods of reaching patients, as well as adeguate 
mechanisms for feedback in the process, were noted as suitable options for enhancing 
consultation in HTAs. 

There was a strong theme of education of patients and the public so that they can be an 
active part of HTAs. 

Education of patient groups and committees was essential in the creation of active and 
informed agents in medicine HTAs. 

Australia, Canada, and the UK are rising to the challenge [of public involvement]; much 
can be learned from the efforts of agencies in these three countries. However, more 
research, funding, education, and HTA process changes are reguired to shift patients and 
the public towards the center of the decision-making model. 

An educated public can provide guality information on citizen's values, needs, judgments, 
prioritization and technology preferences. 
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A015 httos://dr BMC Involvina oatient in the 

ive.aooal Health earlv staaes of health 

e.com/oo Service technoloav assessment 

en?id=0B Research (HTA): a studv protocol 

wOTUaaz 
NAv5b3h 
U02JWS 
1owTm8 

The evaluation of public involvement in HTA prioritization in the UK has demonstrated the 
importance of providing dedicated staff and regular feedback to support this 
involvement. Therefore, patients will receive training in HTA and evidence-informed 
healthcare before the prioritization meeting. A member of the research team will also 
provide ongoing support for patient representatives throughout the process. 

A033 httos://dr Consum Consumer Particioation 
ive.aooal er Health in the Review of Health 
e.com/oo Forum of Technoloav Assessment 
en?id=0B Australia Report on outcomes of 
wOTUaaz consumer consultations 

NAv5SXU 
vMUsvdF 
N4VE0 

Key recommendations of the Consumer Health Forum report were: 

Extensive consumer education to ensure consumers understand medical devices and 
technologies and the system designed to regulate their use 

Consumer education programs should be developed to ensure that consumers have a 
strong understanding of both medical devices and technologies and the HTA system 
designed to regulate their use in Australia. 

Consumers suggested that consumer education should be an integral function of the 
new HTA system 

Consumers strongly endorsed widespread consumer education on health technologies 
and the HTA process, so that consumers: 

Know about the elements of the HTA system, including compliance requirements, 
Understand the benefits and risks of using (or not using) various devices - ensuring 
complaints mechanisms, and where to report adverse events - enabling consumers to 
feed into the HTA system and ensuring learnings from current patient experiences (about 
what works and what does not) and the patient"s journey are taken into account in the 
assessment and review processes. 

Long-term education should begin in schools, and emphasise the rights of all health 
consumers. 

Consumer education is insufficient, and that there should also be education of health 
professionals. Health literacy programs could enable them to communicate better with 
health consumers about health technologies. This would improve consumers" ability to 
provide informed consent when their health professionals are proposing the use of a 



technology in their treatment. 

Consumers indicated the need for education about adverse incident reporting to support 
shared decision making on the use of health technologies. 

A037 httos://dr Expert Role of patient and public 

ive.aooal Reviews participation in health 

e.com/oo Ltd technoloav assessment 

en?id=0B and coveraae decisions 

wOTUaaz 
NAv5UER 
OckEwbl 
96VEE 

Training of patient or public representatives on committees: to ensure that patient or 
public representatives feel able to contribute meaningfully to discussions, there is a need 
for educational opportunities that introduce them to basic terms, concepts and policy 
options; 

NA htto://we Website Patient and public 

b. archive. involvement policv 

ora/web/ 

2015030 

4054755/ 

htto://ww 

w.nice.or 

a.uk/abo 

ut/nice- 

communi 

ties/oubli 

Ci 

involvem 

ent/oatie 

nt-and- 

public- 

involvem 

ent-policv 

NICE 'offer support and training to lay people who contribute to NICE'S work' 
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A checklist of questions to ask at the start of public 
involvement in HTA 


This short list of questions has been collated from knowledge gained from the literature 

review and was written by Jack Nunn and Jo Root. The questions are designed to stimulate 

the creation of a robust and transparent model of public involvement in the HTA process. 

Why do the public need to be involved? 

• Are there any exclusions, are there certain people you don't want to involve? Why? 

• Are there any people who might need additional support to be involved? What support 
might they need? 

What stage of your process are you planning to involve the public? 

• Are you planning to involve them from the start? If not, why? 

• Will you involve the public in helping you plan how you will involve the public? 

• Do you want different people involved at different stages? Why? 

How will you involve the public? 

• Have you involved the public in planning how you will involve the public? For example, 
have you asked members of the public if they think your plans may unintentionally 
exclude some people? 

• Have you got adequate resources or is there a limit to your involvement capacity? 
Have you been transparent about this limit? 

How will you let people know what you are doing? 

• Will the public be involved in helping you plan how you will let people know how to be 
involved (for example, recruitment or dissemination) 

How will you evaluate your public involvement? 

• Have you planned adequate resources to evaluate involvement, including a longer 
term impact assessment? 

• Will you publish or share the results of your evaluation so other may share in any 
learning. 


Appendix 


Search report 
Method 

Search strategies included searching: 

• Peer reviewed journals (including using MEDLINE and library search engines) 

• Grey literature (including Government papers, NGO literature, academic literature, 
relevant websites) using search engines such as 'Google' 

• Social media (a search of social media for key terms, including blogs) 

In addition, a number of interviews were conducted with international subject experts. 

Owing to time and language-constraints, the search only included English language searches. 

The three main categories of the search were (please note '/' denotes the 'OR' boolean 
operator): 

• Health technology AND patient/consumer/public/community/citizen 
involvement/participation/engagement 

• Health technology assessment AND patient/consumer/public/community/user/citizen 
AND involvement/participation/engagement 

• Models/framework AND patient/consumer/public/community/user/citizen AND 
involvement/participation/engagement 

Please note that in the 'Full data extraction output', a record was kept of how each individual 
document was found, and where those search terms were entered. 
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Summary of search 


Search terms 

Database/ 
Website URL 

Documents 

searched 

(number) 

Documents REF (list all docs 
in this search) 

Health technology assessment 

Latrobe search 
engine 

18 

A001.A010, A016.A017, 
A01 8, A01 9, A020, A021, 
A022, A023, 

A024, A025, A026, A027, 
A028, A029, A030, A003 

Health device 

Latrobe search 
engine 

5 

A004, A01 2, A01 3, A014, 
A01 5 

Health technology assessment, 
subcategory 'patient participation 1 

MEDLINE 

1 

A005 

Health technology assessment 

MEDLINE 

3 

A006, A008, A01 1 

Consumer involvement in health 
technology 

Latrobe search 
engine 

1 

A007 

Health technology assessment 
subcategory 'organisations & 
Administration 1 

MEDLINE 

1 

A009 

health technology assessment 

Google 

5 

A031 , A032, A022, A034, 
A035, A041 

Public engagement in health 
technology assessments 

Latrobe search 
engine 

1 

A002 

Interview with experts 


4 

A036, A037, A038, A039 

Health Technology Assessment AND 
development AND training 

CINAHL 

2 

A044, A045 

Health technology assessment 
government 

Google 

1 

A041 


Exclusion criteria 

As this project involved a 'rapid literature review', rather than, for example, a 'systematic 
review' a number of decisions were made in order to balance efficiency and effectiveness. 

Regarding inclusion and exclusion criteria, a number were proposed before the search began, 
but owing to the paucity of literature, the results suggested a simple date range exclusion 
would be the most effective. As a result, anything before the year 2000 was excluded, as this 
date represented the time period that relevant literature began to emerge. 

Some documents were deemed 'out of scope' and not relevant. Records were kept of which 
documents were excluded and why. 

Documents published by MBS were not included in the search but are referenced in the report 
where appropriate. 


Please note, this search only included English language documents. 
Exclusions were: 

• Must be after 2000 

• Must be in English 


Appraisal of the literature review and report 

The following points are a short appraisal of the process of creating this literature review and 

report: 

Limited time: 

• As this project involved a 'rapid literature review', rather than, for example, a 
'systematic review' a number of decisions were made in order to balance efficiency 
and effectiveness. 

• The literature search and data extraction was completed in 49 hours (this does not 
include the time taken to synthesise data and the creation of this report). 

• Exclusion criteria were applied retrospectively, based on the findings. 

Possible publication bias: 

• The literature review returned a limited amount of literature on the subject, in relation 
to what might be expected considering the number of organisations involved in this 
work. This suggests there may be an inherent publication bias, or other factors 
influencing the paucity of literature on this subject. Further research would be required 
to answer this question. 

Lack of systematic evaluation: 

• The literature search found no apparent validated methods for evaluating models of 
involving the public and the impact these might have, making it improper to report on 
any preferred methods. 
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Literature review - Data extraction output - Summary 

This section contains a summary of the data extracted from the literature search. The full data extraction output can be accessed in the following 
ways: 


Document type with 

Microsoft 

Comma-seoarated 

Portable 

Gooale Doc 

link 

Excel 

values (CSV') 

Document 
Format (PDF') 



Please note 'REF' refers to the unique reference number assigned to each document during the literature search. 


REF 

Link 

Issuing Entity 

Title 

Type / 
media 

Ye 

ar 

Summary 

Key learning points or recommendations 

A001 

httos 

Health Policy 82 

Bringing 'the 

Journal 

20 

This paper offers a 

• There is a need for greater public involvement in HT 


://dri 


public' into 


07 

framework that clearly 

and HTA's in Canada. 


ve.ao 


health 



distinguishes specific 



ogle. 


technology 



roles for the public, 

• HTA and HT policy organizations face decisions 


com/ 


assessment 



and relates them to 

about whether to approach public involvement in an 


ooen 


and coverage 



several layers of policy 

ad-hoc or more institutionalized manner 


?id=0 


policy 



analysis and policy 



BvvG 


decisions: 



making where 'the 

• Stakeholder involvement presented as public 


FVO 


From 



public' may engage in 

involvement gives greater voice to professionals and 


F01 A 


principles to 



different tasks. The 

industry interests than to citizens and patients. 


MNX 


practice 



framework offers a 

Moreover, when "public” and "stakeholders" both sit at 


ZOM 





menu of choices for 

the table, inequalities in their powers of persuasion 


XZD 





policy makers 

must be overcome if the public perspective is to have 


Okhl 





contemplating 

any force. 


Ml U 





changes to public 








involvement, as well 

• Discussion points: Public involvement models, 







as a model that can be 

choosing among models and mechanisms, who to 







used to characterize 

involve, accountability, committee representation 









and analyze different 
approaches across 
jurisdictions. 

"Governments are politically accountable for their 
resource allocation decisions 

Canadian policy makers are at an early stage in the 
design of legitimate mechanisms for the public to 
contribute to, and to be apprised of, HTA and 
coverage decisions 

As they consider the options, questions arise about 
whom to involve (e.g, which publics), how to engage 
them (e.g., through what public involvement or 
accountability mechanisms), and for what purpose 
(e.g., to inform the public of decisions and their 
rationales, or to have the public directly affect those 
decisions). 

The difference between public accountability and 
public participation, are not well articulated or 
distinguished in these debates." 

A002 

htto:/ 

BMC Medical 

Public 

Journal 

20 

This study will 

• HTA organizations regularly make decisions about 


/web 

Informatics and 

preferences 


15 

examine public 

when and how public engagement should occur but 


.arch 

Decision Making 

for 



preferences for 

without consideration of the public's preferences on 


ive.or 


engagement 



engagement in 

the method and extent of engagement 


a/we 


in Health 



Australian HTA 



b/20 


Technology 



decision-making using 

• Public engagement is a means to achieve these 


1511 


Assessment 



an exploratory mixed 

aims [of HTA], because it increases the transparency 


0606 


decision- 



methods design. 

of the process and provides a mechanism to consider 


3004 


making: 




social values related to the technology being 


/htto: 


protocol of a 



The objectives of this 

assessed 


//ww 


mixed 



study are to determine 



w.bio 


methods 



if and how the public 

• In HTA, public engagement methods can include 


med 


study 



would like their views 

basic information provision, to consultative 
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centr 

al.co 

m/co 

ntent 

/pdf/ 

si 29 

lb 

015- 

0176 

O.pdf 





and preferences to be 
considered in HTAs 
and whether this 
differs depending 
upon the question 
posed by decision- 
makers and/or the 
characteristics of the 
HTA 

mechanisms such as focus groups and consumer 
representation on decision making committees, to 
more complex methods of deliberation including 
citizens juries and consensus panels. 

• Understanding the effect of a decision involves 
issues such as the number of people with the 
condition, their characteristics (e.g. age, ethnicity), the 
perceived benefit of the technology (relative to the 
severity/burden of the condition), the availability of 
other treatment options, whether expert opinion is 
divided on the effectiveness of the technology and 
the cost of the technology 

• Internal decision-making context factors, on the 
other hand, are factors relating to the decision- 
making process. Such factors include the time and 
resources available to the HTAO, the organizational 
cultural of the HTAO. These factors are rarely made 
explicit in public engagement frameworks, but have 
been noted as being just as influential in determining 
when and how members of the public would be 
engaged. 

• None [of the studies analysed] have specifically 
considered the views of the general public 

• Policy makers [need to] to clarify what they want 
from the public in these circumstances [20] including 
the extent to which they are willing to cede or not, to 
the public's views. 

A003 

httos 
:// dri 

BMC Health 
Service Research 

Methods of 
international 

Journal 

20 

13 

One component of 
HTAs are economic 

There is a considerable unexplainable variance in 
recommendations. Further effort is needed to 



ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5dTlr 

NGJ 

LYjBt 

Q3c 


health 
technology 
assessment 
agencies for 
economic 
evaluations- a 
comparative 
analysis 



aspects. To 
incorporate economic 
aspects commonly 
economic evaluations 
are performed. The 
objective of this paper 
is to provide an 
overview and 
comparison of the 
methodological 
recommendations of 
international HTA 
agencies for economic 
evaluations through 
comparative analysis 
research. 

harmonize methods for preparing economic 
evaluations 

A004 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5NW 

9tZG 

pYV 

mN3 

bHM 

European Heart 
Journal 

Health 

technology 

assessment 

in 

interventional 
electrophysiol 
ogy and 
device 
therapy: a 
position 
paper of the 
European 
Heart Rhythm 
Associationt 

Journal 

20 

13 

This position paper 
reviews a series of 
issues that are related 
to HTAs for catheter 
ablation and device 
therapy and that in our 
view deserve attention 
(economic 
evaluations, 
organisational and 
educational issues, 
sensitivity analyses). 

THIS RECOMMENDATION RELAES TO CE IN HTA's: 
The scoping, assessment, and appraisal of an 
emerging therapy should address questions that are 
relevant to patient groups in which clinical 
effectiveness has been proved. As outlined in clinical 
practice guidelines, such patient groups should be 
defined in collaboration with experienced clinicians. 

A005 

httos 

://dri 

International 
Journal of 

Perspectives 
on health 

Journal 

20 

04 

This paper laments 
that those responsible 

Engagement of patients and citizens ought to be 
encouraged at all stages in the HTA process, in 
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ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5ZT 

BIZz 

VCc 

m9C 

b2s 

Technology 
Assessment in 
Health Care 

technology 

assessment: 

response 

from the 

patient's 

perspective 



for HTA's should make 
greater efforts to 
involve the public and 
ensure that the 
findings of HTA’s are 
accessible to patients 
for use when making 
treatment choices. 
Patients and the public 
make a valuable 
contribution to HTA’s, 
yet the findings are 
not always accessible 
to them. 

particular the following: 

• Determining priorities for assessment. 

• Designing and conducting assessments and 
appraisals 

• Receiving and using the findings from HTA. 

• Engaging in debates about policy priorities and 
rationing 

• Involving patients in the design of trials or reviews 
and choice of outcome measures can greatly 
increase the relevance of the final product 

• Involving patients in the design of trials or reviews 
and choice of outcome measures can greatly 
increase the relevance of the final product 

• What is needed is a systematic attempt to engage 
the views of citizens, to balance those of the specific 
interests. 

•Those responsible for health-care budgets must 
ensure that individual clinical decisions fit within the 
wider context of resource availability and public 
priorities. 

•Clinical freedom and patient choice cannot be 
divorced from its societal context, and when 
individual expectations clash with population 
priorities, there must be a mechanism for resolving 
the conflict interest groups 

A006 

httos 
:// dri 
ve.ao 

BMJ Publishing 
Group 

NICE 

supports 

allowing 

Letter 

20 

14 

Rebutal to the article 
by Wise claiming 
that "selling” scientific 

Scientific advice is an essential tool for enabling 
constructive and structured dialogue between HTA 
decision making bodies and industry to ensure that 




ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5V2F 

wbD 

h6M 

TIsZ 

Gc 


technology 
assessment 
bodies to 
provide 
advice to 
drug industry 



advice procedures 
enables institutional 
capture by drug 
companies. 

product development plans meet the requirements of 
decisions makers. In this way, scientific advice can 
facilitate the development of clinically effective, 
useful, and affordable medicines for the benefit of 
patients. 

A007 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5V2F 

wbD 

h6M 

TIsZ 

Gc 

Health Policy 69 

A consumer 

involvement 

model for 

health 

technology 

assessment 

in Canada 

Journal 

20 

04 

• Health consumers 
believe they should 
have a role in 
determining their 
treatment options. 

• By providing support 
and training, these 
organizations believe 
they would be capable 
of and are willing to 
devote a considerable 
amount of time and 
effort to have an 
influence on the 
assessment and 
evaluation of potential 
treatments and 
therapies and their 
availability for access. 

• In order to have a fair and transparent process, an 
independent consumer led organization dedicated to 
health technology assessment should be created and 
supported by the national government. 

• This PITA consumer organization would be 
responsible for developing a network with health 
organizations, the development of a database of 
health consumers' expertise, knowledge, etc, the 
development of a formal health consumer 
stakeholder involvement process, the training and 
education of health consumers, information 
dissemination, and the evaluation of the processes 
and program. 

• These activities would provide this organization the 
support and resources needed to nominate the most 
effective health consumers to participate in HTA 
committees. 
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• This model also requires the support of the central 
organization conducting HTA reviews to respect the 
health consumer stakeholder involvement process 
and provide resources for accommodating the needs 
of these health consumer experts 

A008 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5bjJ 

LT09 

OSF 

NXR 

IA 

American 
Association for 
Cancer Research 

The Health 
Technology 
Assessment 
of 

Companion 
Diagnostics: 
Experience of 
NICE 

Journal 

20 

14 

Companion 
diagnostics are used 
to aid clinical decision 
making to identify 
patients who are most 
likely to respond to 
treatment. They are 
becoming increasingly 
important as more 
new pharmaceuticals 
receive licensed 
indications that 
require the use of a 
companion diagnostic 
to identify the 
appropriate patient 
subgroup for 
treatment. These 
pharmaceuticals have 
proven benefit in the 
treatment of some 
cancers and other 
diseases, and also 
have potential to 
precisely tailor 
treatments to the 
individual in the future. 
However, the 

This article could provide some insights into broader 
considerations, however i think it is largely OUT OF 
SCOPE. 


increasing use of 
companion 
diagnostics could 
place a substantial 
burden on health 
system resources to 
provide potentially 
high volumes of 
testing. This situation, 
in part, has led policy 
makers and Health 
Technology 
Assessment (HTA) 
bodies to review the 
policies and methods 
used to make 
reimbursement 
decisions for 
pharmaceuticals 
reguiring companion 
diagnostics. The 
assessment of a 
pharmaceutical 
alongside the 
companion diagnostic 
used in the clinical 
trials may be relatively 
straightforward, 
although there are a 
number of challenges 
associated with 
assessing 
pharmaceuticals 
where a range of 


This document was created by Jack Nunn for the Consumer Health Forum on 1 1 October 201 6. To give feedback, send an email or contact fSJackNunn . 


Ill 








alternative companion 
diagnostics are 
available for use in 
routine clinical 
practice 


A009 

httos 

BMC Medical 

Technology 

Journal 

20 

With a growing 

Findings suggest that largely local and relatively ad 


:// dri 

Ethics 

assessment 


09 

number of genetic 

hoc decision making processes are being made in 


ve.ao 


and resource 



tests becoming 

relation to resource allocations for predictive genetic 


ogle. 


allocation for 



available to the health 

tests and that a more coordinated and, potentially, 


com/ 


predictive 



and consumer 

national approach to allocation decisions in this 


ooen 


genetic 



markets, genetic 

context may be appropriate 


?id=0 


testing: A 



health care providers 



BwO 


study of the 



in Canada are faced 



lUaa 


perspectives 



with the challenge of 



zNAv 


of Canadian 



developing robust 



5RW 


genetic 



decision rules or 



xaYO 


health care 



guidelines to allocate 



IrSzc 


providers 



a finite number of 



xYnc 





public resources. The 
objective of this study 
was to gain Canadian 
genetic health 
providers' 
perspectives on 
factors and criteria 
that influence and 
shape resource 
allocation decisions 
for publically funded 
predictive genetic 
testing in Canada 


A010 

httos 

BMC Health 

Study 

Study 

20 

Is only a study 



://dri 

Service Research 

Protocoflntro 

Protocol 

09 

protocol- would be 





ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5cOR 

6a3l 

1 Y19 

acm 

c 


ducing 

patient 

perspective in 

health 

technology 

assessment 

at 

the local level 



good to see if the 
study has taken place. 
This study provides an 
opportunity to bridge 
the gap between HTA 
producers 

and its ultimate end- 
user: the patient. It will 
provide guidance to 
support local HTA 
units in Quebec and 
elsewhere in 
their decisions 
regarding patient 
participation. The 
framework developed 
could be applied to 
design and implement 
strategies for involving 
patients in HTA 
activities. 


A011 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5cnZ 

WV2 

BMJ Publishing 
Group 

European 
Medicines 
Agency is 
attacked over 
proposal to 
allow 

technology 
assessment 
bodies to sell 
advice to 
drug industry 

News 

20 

14 

The European 
Medicines Agency has 
been criticised over its 
plan to allow health 
technology 
assessment bodies to 
provide confidential 
advice to drug 
companies in 
exchange for fees. A 
coalition of medicine 
advocates denounced 

Again a broader consideration that you may or may 
not tie into your work. 
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xlV2 

h4W 

HM 





the practice, which 
they said was a clear 
conflict of interest. 


A012 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5LU 

MxM 

09pV 

XRiQ 

kU 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Survey on the 

involvement 

of consumers 

in health 

technology 

assessment 

programs 

Research 

Report/ 

Journal 

20 

06 

The aim of this study 
was to obtain 
information from 
members of the 
International Network 
of Agencies for Health 
Technology 
Assessment (INAHTA) 
on their involvement 
of consumers 
(patients, carers, and 
related organizations) 
in their programs 

Of the thirty-seven agencies that provided responses, 
57 percent involve consumers in some aspects of 
their HTA programs and 83 percent intend to involve 
consumers in the future. Summaries of HTA reports 
that are intended to be easily understood by 
consumers are prepared by 49 percent of the 
agencies, and 36 percent involve consumers in 
dissemination of HTA material. 

A013 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5NG 

1 Vd2 

tFUk 

ZlYk 

k 

BMC Public 
Health 

Medical 
device 
assessment: 
scientific 
evidence 
examined by 
the French 
national 
agency for 
health - a 
descriptive 
study 

Journal 

20 

12 

This study aimed to 
ascertain level of 
evidence available for 
implantable medical 
devices (IMDs) access 
to reimbursement in 
France. 

This study confirmed that level of evidence of clinical 
evaluation of implantable medical devices (IMDs) is 
low and needs to be 
improved 


A014 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5enJ 

kSHG 

N2Zr 

RVU 

BMC Health 
Service Research 

Involving 
patients in 
HTA activities 
at local level: 
a study 
protocol 
based on the 
collaboration 
between 
researchers 
and 

knowledge 

users 

Study 

Protocol 

20 

12 

Study objectives are 
to: 1) validate a 
reference framework 
for exploring the 
relevance and 
applicability of various 
models of patient 
involvement in HTA, 2) 
implement strategies 
that involve patients 
(including close 
relatives and 
representatives) at 
different stages of the 
HTA process, 3) 
evaluate intervention 
processes, and 4) 
explore the impact of 
these interventions on 
a) the applicability and 
acceptability of 
recommendations 
arising from the 
assessment, b) patient 
satisfaction, and c) the 
sustainability of this 
approach in HTA. 

This project represents a unique opportunity for 
policy makers and researchers to work toward a 
common goal to test different forms of patient 
involvement in HTA and to evaluate them in order to 
produce useful knowledge regarding the 
implementation of patient involvement strategies in 
HTA and their possible effects and impacts. 

A015 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

BMC Health 
Service Research 

Involving 
patient in the 
early stages 
of health 
technology 
assessment 

Study 

Protocol 

20 

14 

The project objectives 
are: 1) setting up 
interventions to 
promote patient 
participation in three 
stages of the HTA 

Assessing the impact of patient participation on the 
relevance of the topics suggested, the prioritization 
process, and the assessment plan from the point of 
view of patients and other groups involved in HTA. 
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?id=0 

BwQ 

TUqa 

zNAy 

5b3li 

uQ2J 

WS1 

pwT 

m8 


(HTA): a 

study 

protocol 



process: identification 
of HTA topics, 
prioritization, and 
development of the 
assessment plan of 
the topic prioritized; 
and 2) assessing the 
impact of patient 
participation on the 
relevance of the topics 
suggested, the 
prioritization process, 
and the assessment 
plan from the point of 
view of patients and 
other qroups involved 
in HTA. 


A016 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5VXh 

DTUt 

QZE 

8zc 

WM 

Social Sciences & 
Medicine 

"It all 

depends”: 
Conceptualizi 
ng public 
involvement 
in the context 
of health 
technology 
assessment 
agencies 

Journal 

20 

10 

The concept of public 
involvement is poorly 
articulated and little 
attention has been 
paid to the context of 
HTA agencies. This 
article investigates 
how public 
involvement is 
conceptualized in the 
HTA agency 
environment. Using 
qualitative concept 
analysis methods, we 
reviewed the HTA 
literature and the 

Our analysis reveals that HTA agencies' role as 
bridges or boundary organizations situated at the 
frontier of research and policymaking causes the 
agencies to struggle with the idea of public 
involvement. The HTA community is concerned with 
conceptualizing public involvement in such a way as 
to meet scientific and methodological standards 
without neglecting its responsibilities to healthcare 
policymakers. 








websites of HTA 
agencies and 
conducted semi- 
structured interviews 
with informants in 
Canada, Denmark, and 
the United Kingdom. 


A017 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

open 

?id=0 

BwQ 

TUqa 

zNAy 

5c1 R 

VLV 

VERj 

Znb 

Gs 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Consumer 

involvement 

in the health 

technology 

assessment 

program 

Journal 

20 

04 

This study aims to 
describe a cycle of 
development leading 
to sustainable 
methods for involving 
consumers in the 
management of a 
program 

commissioning health 

technology 

assessment 

Over 4 years, procedures and resources have been 
developed to support six consumers attending seven 
to eight prioritization meetings a year; thirty to forty- 
five consumers each year commenting on research 
need for particular topics; thirty consumers a year 
commenting on research proposals, and twenty a 
year commenting on research reports. The 
procedures include clear job descriptions, induction 
and development days, clear briefing materials, 
payment for substantial tasks, and regularly seeking 
feedback to improve procedures 

A018 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5Rkx 

Social Sciences & 
Medicine 

Eliciting 
ethical and 
social values 
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Our primary objective 
was to elicit a set of 
ethical and social 
values from citizens 
that could be used to 
guide Ontario's HTA 
evidentiary review and 
appraisal process. A 
secondary objective 
was to explore the 
feasibility of using 

Over the course of five meetings, panel members 
progressed toward the identification of a set of core 
values e universal access, choice and quality care. 
These values were consistently prioritized as the core 
values that should be considered in the evaluation of 
health technologies and ensuing recommendations. 
Sustained and deliberative methods, like a citizens' 
panel, offer a promising approach for eliciting ethical 
and social values into HTA. 
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participatory 
approaches to elicit 
these values. 
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Discussion 
surrounding the case 
of HTA's in Canada 
over the past 20 years. 
Highlighting 
shortcomings, 
challenges and 
considerations about 
the future. 

It is difficult, if not impossible, for a single HTA body 
to meet the information needs of decision-makers 
unless it is operating within a single payer system, 
such as the United Kingdom. In Canada, HTA has 
evolved to include a combination of national and local 
initiatives, reflecting the decentralized nature of its 
health-care system 
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This article develops a 
framework to help 
decision makers 
supplement the 
standard efficiency 
criteria of HTA and 
avoid building 
inequities, explicit or 
implicit, into their 
methods. The 
framework is intended 
as a first step toward 
creating a checklist for 
alerting decision 

Two domains of equity are especially relevant in HTA. 
One is fairness of the procedures used in the conduct 
of HTAs. The other is equity as a decision criterion, 
like efficiency, for ranking health care interventions. 

We propose a practical and adaptable initial 
framework as the basis for the development of a 
more comprehensive typology (Note 4). It is practical 
because it is intended as a sequence of red flags to 
alert decision makers— and the designers of the 
systems within which they work— to matters of equity 
that might warrant integration into, or at least parallel 
consideration with, the usual efficiency analysis of 
HTAs. It is adaptable because the framework as it 
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makers to a wide 
range of equity 
considerations for 
HTA. This framework 
is intended be used as 
part of the process 
through which 
advisory bodies 
receive their terms of 
reference; scope the 
agenda prior to the 
selection of a 
candidate intervention 
and its comparators 
for HTA; prepare 
background briefing 
for decision makers; 
and help to structure 
the discussion and 
composition of 
professional and lay 
advisory groups 
during the 

assessment process 

currently stands is intended only as an initial step and 
what may be added is currently unknowable (at least, 
by us). 
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The main contribution 
of this article is to 
show that a dialogue 
between ethics and 
economics, prior to an 
assessment, makes it 
possible to redefine 
the choice of 
effectiveness criteria 

• The congenital nature of the two disabilities and the 
irreversible consequences of both GH treatment and 
cochlear implants leads to envisaging the existence 
of a moral dilemma arising from possible conflicts of 
preferences: the choice of proxy respondents who 
must estimate the value of health benefits provided 
by the two technologies could indeed have a 
determining impact on the results of the assessment, 
and hence on the funding of treatment by health 
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(subjective well-being, 
capabilities or social 
outcomes), the choice 
of perspective 
(patients or the able- 
bodied), as well as the 
scope of assessment 
(medical and non- 
medical care). 

insurance. 

• The main contribution of this article is to show that 
the dilemma could be resolved via a dialogue 
between ethics and economics, whose importance 
Sen stressed in his essay "On Ethics and Economics”. 

• This dialogue indeed makes it possible to redefine 
the choice of perspective (patients or the able- 
bodied), as well as the perimeter of assessment 
(medical and non-medical care). 
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in Health 
Technology 
Assessments 
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Patient and public 
participation in health 
technology 
assessment (HTA) of 
medicines has been 
cited as an important 
component of the 
decision making 
structure; however, 
how to actually 
achieve meaningful 
involvement is less 
understood 

• The HTA process is becoming increasingly 
transparent to patients and the public; however, more 
effort is required to fully engage patients in the 
decision-making processes for medicine HTAs. 

• This pilot study identified three key areas for further 
advancement in this field, and recognized a need for 
further research in the areas of measuring the impact 
of patient engagement on decision making in 
medicine HTAs, as well as the best methods to better 
prepare patient advocacy groups through HTA 
education and training. 
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segments of the population. 

• Clinical effectiveness is only half the story of HTA. 
The other half is equity. And on equity-focused health 
technology assessment, the UK performs poorly. 

• Second, health systems have been incorrectly 
excluded from definitions of health technology. 
Health systems are health technologies. Until HTA 
programmes take health systems research more 
seriously, as many low and middle income countries 
are now doing, they will continue to miss important 
opportunities for strengthening and sustaining the 
health of their populations 
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The aim of this study 
was to catalogue ELSI 
content in HTAs of 
GEP tests for breast 
cancer prognosis and 
examine how this 
material is integrated 
in HTAs. (GEP= Gene 
Expression profile 
tests) 

ELSI= Ethical, legal, 
social issues 

Despite a wide variety of important ELSI raised, these 
were rarely explicitly addressed in HTAs. Explicit 
treatment would increase their accessibility to 
decision-makers, and may augment HTA efficiency 
maximizing their utility. This is particularly important 
where complex Personalized Medicine applications 
are rapidly expanding choices for patients, clinicians 
and healthcare systems. 
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The goal of this study 
is to understand how 

We contend that the use and dissemination of HTA 
reports by specific groups could help in widening the 
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and why patient and 
consumer 

organizations use PITA 
findings within their 
organizations, and 
what factors influence 
how and when they 
communicate their 
findings to members 
or other organizations. 

debate around controversial health technologies. The 
implications and opportunities for PTA agencies 
relate to the following: (i) identification of "lay” 
organizations that could help in disseminating results; 
(ii) acknowledgement of a "lay” audience for PTA 
findings; (iii) strategic inclusion of advocacy groups 
during the assessment process for highly 
controversial technologies; and (iv) contribution of 
these organizations to the push efforts of knowledge 
transfer. 

Although PTA findings are often used by the patient 
and consumer organizations, key differences were 
observed in exactly how the four PTA reports were 
used. Three types of use (instrumental, conceptual, 
and symbolic) are reported and illustrated. We 
highlight the importance of the organization's mission 
and knowledge base in explaining the types of use 
observed. 
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This rapid needs 
assessment evaluates 
the provision of PITA in 
the south-west 
peninsula, and its 
scope, content and 
quality. 

Semi-structured 
interviews and 
documentary analysis 
to assess the need for 
PITA were utilsed in 
these studies. 

The quality of literature review in PTAs was variable 
and virtually none considered value for money. 
Informants felt there was insufficient provision of 
local PTAs. Local focus and clinical engagement 
were seen as key to the implementation of appraisal 
decisions, but this was threatened by weak links with 
commissioning and processes to prioritize decisions 
across primary care trusts. 

Local provision of PTAs in the peninsula is at full 
stretch and quality is variable. Lack of consideration 
of value for money and variation in quality in PTAs 
present a potential risk in terms of clinical and 
corporate governance and may leave appraisal 
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decisions open to challenge. Links between 
committees using HTA and with commissioning are 
weak, limiting capacity and undermining financial and 
health benefits. Furthermore, incongruent 
prioritization processes may be leading to unequal 
access to technologies. 
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In the ideal setting, a patient-based HTA would 
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promote patient knowledge by providing access to 
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promote patient 

information and promoting an informed dialogue 
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empowerment and 

between patients and their healthcare professionals. 
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patient-centered care. 

To implement a patient-based HTA, the focus must 
turn to the patient's issues and incorporate each 
patient's unique perspective and preferences. 
Processes must change to increase patient 
participation in all levels of HTA and aim to promote 
empowered patients who can make informed 
decisions 
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funding 

innovative 

health 

technologies? 
The case of 
PET scanning 
in Ontario 



technology is being 
delayed until there is 
sufficient evidence to 
justify a coverage 
decision. The case 
considered is the 
Ontario (Canada) 
government's decision 
to restrict access to 
positron emission 
tomography (PET) 
scans until further 
evidence becomes 
available. 

The case was 
deliberated on by 
twenty-six members 
of the Toronto FHealth 
Policy Citizens' 
Council, with a follow- 
up survey 
administered to 
individual council 
members 

Public support for arrangements that limit access to 
new technologies will likely vary depending on the 
details of the specific arrangement being proposed. 
Deliberative public dialogue can be effectively used to 
identify cases the general public is most likely to 
support 
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The Breaking 
Boundaries review 
was commissioned by 
the director general of 
research and 
development and chief 
medical officer in 
March 201 4 to assess 

The report outlined six strategic goals that the 
institute should work towards for 2025. These include 
ensuring that opportunities to engage with research 
are made visible to the public and seized; that public 
involvement is locally driven but strategically 
consistent; that the public's contribution is valued and 
required as part of high quality research; and that 
evidence of what works is collected and 
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how successful the 
institute has been in 
achieving public 
involvement and to 
make 

recommendations to 
build on the 
achievements so far. 

disseminated. The report also gave 1 1 specific 
recommendations for achieving this vision, central to 
which is that the institute should commission the 
development of a set of values, principles, and 
standards for public involvement. All NIHR leaders, 
funded researchers, and staff should also receive an 
induction in public involvement, it added 
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• Canada has health 
technology 

assessment programs 
at national, provincial 
and local levels. The 
context of HTA's in 
Canada is explored 
and 'lessons for 
Australia 1 are also 
mentioned. 

The programs have been complementary in providing 
advice to decision makers in health care. 

A national strategy for the management of health 
technologies is expected to strengthen 
communication with policy areas. 
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• provide improved opportunities for sponsor and 
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HTA Review is to 
identify opportunities 
for reform of the 
processes that may be 
poorly designed, 
duplicated or 
unnecessary, 
imposing unwarranted 
costs and complexity 
on business and 
discouraging 
innovation. 

stakeholder input, including expanded options for 
seeking review of decisions; 

• operate through a single entry point to assist 
stakeholders by receiving, guiding and monitoring all 
applications for reimbursement; 

• be more flexible in dealing with novel and complex 
technologies by coordinating assessments and 
allowing different aspects of complex technologies to 
be assessed at the same time; 

• speed up Medical Services Advisory Committee 
assessments by allowing sponsors to supply their 
own assessments and have these critiqued; 

• simplify administration of the Prostheses List by 
streamlining administrative processes and removing 
duplication; and 

• reform post-market surveillance of health 
technologies to strengthen patient safety and value 
for money for taxpayers. 
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CHFA touch on 
various facets of HTA. 
There is considerable 
reference to the role of 
the consumer in HTA's 
interwoven throughout 
the paper. 

Consumers are concerned that the current 
assessment processes do not recognise that the 
interaction of what are effectively two or more 
different technologies may have different health 
outcomes to when they are used alone. 

Consumers are also concerned that the current HTA 
arrangements delay the introduction of new 
technologies for what may be years at a time. 

While CHF does not support untested technologies 
being introduced to the Australian market, the current 
time taken for assessment for some products 
appears unnecessarily long. For those consumers in 
particular who are managing a terminal condition, this 









is unacceptable. 

With respect to the cost-effectiveness of HTA, some 
consumers are not convinced that the plethora of 
'substantially equivalent' technologies currently listed 
and subsidised by the Australian Government is 
justified. 

Consumer observations and experiences of health 
technology need to be included in post-marketing 
surveillance of medical devices and medicines in a 
systematic way. Consumers are very concerned that 
their accounts of adverse events are not regarded as 
credible within the health industry. This is a serious 
public health issue, as it is consumers who are using 
health technology who are most likely to observe and 
report any ill effects. 

CHF consultations indicate that key issues for 
consumers with respect to co-dependent and hybrid 
technologies include (i) being able to provide 
expertise in evaluating these technologies; (ii) having 
the infrastructure and processes that can handle the 
information sources and data management from 
various evidence sources; and (iii) having strong 
national and international collaborative networks so 
that expertise can be drawn from all over the world 

A033 

httos 

:// dri 

ve.ao 

ogle. 

com/ 

ooen 

Consumer Health 
Forum of 
Australia 

Consumer 
Participation 
in the Review 
of Health 
Technology 
Assessment 

Report 

20 

09 

The report is written 
from a consumer 
perspective, and 
specifically outlines 
consumer opinions, 
concerns and 

Recommendation 1 : 

Robust mechanisms should be developed to take into 
account consumer experiences and needs in all HTA 
stages through mechanisms such as: o Consumer 
representatives on committees; o Use of consumer 
impact statements; o Public reporting of consumer 
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experiences with devices and technologies; and o 
International models as appropriate (eg Citizens" 
Councils, Interest SubGroups) 

Recommendation 2: 

Rigorous post market surveillance mechanisms 
should be established, ideally governed by a separate 
body, to the body conducting assessments, to enable 
the capture, analysis and reporting of information on 
adverse events and for this information to feed back 
into assessment processes. 

Recommendation 3: 

The FITA system should have the capacity to reduce 
red tape, bureaucracy and inefficiencies in the 
system, including through a single entry point into the 
system and a process for interim approval for some 
devices, to ensure that they are available to 
consumers more quickly, pending full assessment. 
Recommendation 4: 

FITA pre-market assessment processes must apply 
clear risk criteria in the assessment of evidence for 
applications with consumer safety allocated the 
highest priority. 

Recommendation 5: 

There should be mechanisms for regularly reviewing 
the technologies on the ARTG, and removing them 
from the ARTG when they have been shown to be 
dangerous, when they are superseded by superior 
products, or when they are no longer being used. 
Recommendation 6: Consumer education programs 
should be developed to ensure that consumers have 
a strong understanding of both medical devices and 
technologies and the FITA system designed to 
regulate their use in Australia. 

Recommendation 7: A single rigorous code of 










conduct for all pharmaceutical and therapeutic goods 
industries with appropriate sanctions for non- 
compliance with the code should be established. 
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FITA in Australia now 
has a long history and 
is well established as 
a source of advice to 
health decision 
makers. Challenges 
remain in extending 
the scope of 
assessments, 
developing more 
transparent 
approaches in some 
areas, and 

consistently applying 
appropriate standards. 

Most FTA activity in Australia has been associated 
with provision of advice for the two national subsidy 
programs, Medicare, and the Pharmaceutical Benefits 
Scheme (PBS). National advisory bodies established 
by the federal government have had a prominent role. 
Assessments from the advisory bodies have had a 
major influence on decisions related to Medicare and 
the PBS, and in some other areas. Technologies 
without links to the national subsidy schemes, and 
those that are widely distributed, have been less well 
covered by FTA. To some extent these are addressed 
by evaluations supported by state governments, but 
details of approaches taken are not readily available 
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Discusses the state of 
FITA's in England. Also 
provides commentary 
on the Australian 
context, highlighting 
differences. 

The Fealth Technology Assessment (FTA) 
Programme in England is a government-funded but 
independent research program. 

• It is "needs-led", identifying technologies of most 
importance to the National Fealth Service and 
commissioning research to provide answers on these 
technologies useful to policymakers, clinicians and 
patients. 

• It is "science-added", refining problems to 
researchable questions and working with researchers 
to ensure that the question is addressed, and 
disseminating the findings to key audiences. 
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• There is a clear distinction in England between 
assessment (a scientific process and the role of the 
HTA Programme) and appraisal (the role of 
policymakers, like the National Institute for Health 
and Clinical Excellence). 
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in-depth interviews 
with top officials at 
these agencies 


A037 

httos 

Expert Reviews 

Role of 

Journal 

20 

In this article, we 

A review of 'grey' literature, key informant interviews 


://dri 

Ltd 

patient and 


11 

summarize findings 

and a workshop of Canadian researchers, decision- 


ve.ao 


public 



from peer-reviewed 

makers and patient representatives with an interest in 


ogle. 


participation 



and 'grey' literature, 

public and patient involvement in coverage decision- 


com/ 


in health 



and discussions with 

making identified several issues related to such 


ooen 


technology 



key informants to 

approaches: 


?id=0 


assessment 



determine potential 



BwO 


and coverage 



roles for patients and 

• Open (public) committee meetings: public and 


lUaa 


decisions 



the public in HTA and 

patient representatives on committees have 


zNAv 





coverage decision- 

expressed discomfort in expressing their views in 



5UE 

ROck 

Ewbl 

96VE 

E 





making. We also 
summarize existing 
roles for both groups 
in jurisdictions 

public, fearing that their views may not always align 
with the public, in its entirety; 

• Training of patient or public representatives on 
committees: to ensure that patient or public 
representatives feel able to contribute meaningfully to 
discussions, there is a need for educational 
opportunities that introduce them to basic terms, 
concepts and policy options; 

• Lack of resources available to patient organizations 
to assist them in completing submissions: many 
patient/carer organizations are not adequately 
resourced to make submissions, which can be 
complex and multidimensional. Consequently, only 
large, well-funded (often by industry) groups may be 
able to exercise this option 

A038 

httos 

://dri 

ve.ao 

ogle. 

com/ 

ooen 

?id=0 

BwQ 

TUqa 

zNAy 

5em 

xzTI9 

OME 

9Wd 

Ik 

International 
Journal of 
Technology 
Assessment in 
Health Care 

Involvement 
of consumers 
in health 
technology 
assessment 
activities by 
INAHTA 
agencies 

Journal 

20 

13 

The survey results 
suggest that there is a 
trend to increased 
involvement of 
consumers by the 
INAHTA agencies in 
their programs but 
that the level of 
involvement remains 
relatively limited. The 
manner of consumer 
participation varies 
between agencies. 

Of the thirty-three agencies that provided responses, 
67 percent involve consumers in some aspects of 
their health technology assessment (HTA) programs, 
compared with 57 percent in 2005. As in the earlier 
survey, most agencies reporting involvement have 
contact with consumer or patient organizations and a 
large minority also involve individual consumers. 
Summaries of HTA reports that are intended to be 
easily understood by consumers are prepared by 84 
percent of the agencies, and 42 percent involve 
consumers in dissemination of HTA material. In both 
areas, there was some increase from the levels 
previously reported 

A039 

htto:/ 

Department of 

Medicare 

Government 

20 

How services can be 



This document was created by Jack Nunn for the Consumer Health Forum on 1 1 October 201 6. To give feedback, send an email or contact fSJackNunn . 


131 



/web 

.arch 

ive.or 

q/we 

b/20 

1511 

1123 

3210 

/http: 

//ww 

w.he 

alth. 

gov, 

au/in 

terne 

t/ma 

in/ou 

blishi 

nq.n 

sf/C 

onte 

nt/M 

BSRe 

view 

Task 

force 

Health 

Benefits 

Schedule 

Review 

Taskforce 

report 

15 

aligned with 
contemporary clinical 
evidence and improve 
health outcomes for 
patients. 


A040 

htto:/ 

Department of 

HTA Policy 

Government 

20 

An overview of the 

The Australian Government accepted two 


/ww 

Health 

framwork 

report 

11 

current Australian 

recommendations in the review, one point which 


w.he 





Government HTA 

included that the policy framework includes a vision, 


alth. 





processes used to 

goal, objectives and principles which provide a 


ggw 





inform decisions 

systematic and consistent approach to HTA, and an 


au/in 





about the registration 

explicit, high-level statement of direction for the 


terne 





of health technologies 

implementation and integration of the current HTA 




t/hta 

/□ubl 

ishin 

q.nsf 

/Con 

tent/ 

polio 

yd 





for use in Australia, 

functions to form a coherent system. 

A041 

httos 

:// w 

ww.a 

delai 

de.e 

du.a 

u/aht 

a/pu 

bs/A 

HTA 

_fact 

shee 

t_up 

date 

d_Jul 

y_20 

15.p 

df 

University of 
Adelaide 

Adelaide 

Health 

Technology 

Assessment 

(AHTA) 


20 

11 

In a systematic 
manner AHTA 
identifies, collates, 
appraises, and 
synthesises evidence 
on different medical 
services to inform 
policy makers and 
health professionals 
on whether an 
intervention is 
effective and safe. 
Economic modelling is 
conducted to allow an 
assessment of the 
cost-effectiveness of 
these medical 
services. 


A042 

http:/ 

/ww 

w.ef 

pia.e 

u/uol 

oads 

/Mo 

Chalres River 
Associates for 
EFPIA 

A 

comparative 
analysis of 
the role and 
impact of 
Health 
Technology 

Research 

Report 


A comparative 
assessment of the 
role and impact of 
Health Technology 
Assessment (HTA) in 
different parts of the 
world, for example 

nbj 


This document was created by Jack Nunn for the Consumer Health Forum on 1 1 October 201 6. To give feedback, send an email or contact @JackNunn . 


133 



dules 

/Doc 

ume 

nts/h 

ta- 

com 

paris 

on- 

reoor 

t 

uoda 

ted- 

iuJy: 
26 : 
2011 

stc.p 

df 


Assessment 



how different systems 
use HTA, the basis of 
the approach they 
apply, how it works in 
practice and the 
conseguences for the 
key stakeholders. 


A043 

http:/ 

/stm. 

scien 

cem 

ag.or 

g/co 

ntent 

/scitr 

ans 

med/ 

7/31 

3/31 

3ed1 

2.fu II 

.odf 

Science 

Translational 

Medicine 

Hearing 
Voices: FDA 
seeks advice 
from patients 

Editorial 

20 

15 

New patient 
engagement advisory 
committee set up by 
FDA CDRH to advise 
FDA on issues relating 
to the regulation of 
medical devices and 
their use by patients. 

Advisory Model- patients work hand in hand with 
regulators and stakeholders. Engage patients at the 
same time as the emergence of medical device 
innovation, integration of digital health technologies, 
new models for clinical trials, and patient 
engagement 


A044 

http:/ 

/ww 

w.sci 

ence 

direc 

t.co 

m.ez 

.libra 

ry.lat 

robe. 

edu. 

au/s 

cienc 

e/arti 

cle/p 

ii/SO 

1688 

5101 

2002 

813 

Health Policy 

Diffusion and 
use of health 
technology 
assessment 
in policy 
making: 

What lessons 
for 

decentralised 

healthcare 

systems? 

Journal 

Article 


There is probably need 
to rethink the multi- 
layer organizational 
framework of HTA in 
Italy by leveraging on 
current knowledge and 
efficient redistribution 
of activities across 
regions. We would 
advise for different 
jurisdictions playing 
different roles while 
achieving similar 
health outcomes for 
their patients, rather 
than jurisdictions 
aiming at doing 
exactly the same 
things resulting in 
unequal access to 
healthcare service 
provision. 


A045 

htto:/ 

Health 

Avoiding and 

Journal 

20 

This study seeks to 

Published definitions of overall model validity 


/ww 

Technology 

identifying 

Article 

10 

understand the nature 

comprising conceptual model validation, verification 


w.nc 

Assessment 

errors in 



of errors within HTA 

of the computer model, and operational validity of the 


bi.nl 


health 



models, to describe 

use of the model in addressing the real-world problem 


m.ni 


technology 



current processes for 

are consistent with the views expressed by the HTA 


Jigo 


assessment 



minimising the 

community and are therefore recommended as the 


v/bo 


models: 



occurrence of such 

basis for further discussions of model credibility. 


oks/ 


qualitative 



errors and to develop 

Discussions of model credibility should focus on 


NBK 


study and 



a first classification of 

risks, including errors of implementation, errors in 


5684 


methodologic 



errors to aid 

matters of judgement and violations. Discussions of 


6/od 


al review 



discussion of potential 

modelling risks should reflect the potentially complex 


This document was created by Jack Nunn for the Consumer Health Forum on 1 1 October 201 6. To give feedback, send an email or contact @JackNunn . 


135 



f/Bo 

oksh 

elf_N 

BK56 

846. 

gdf 





strategies for avoiding 
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network of cognitive breakdowns that lead to errors in 
models and subseguent failures in decision support. 
Existing research concerning the cognitive basis of 
human error should be brought into the examination 
of modelling errors. There is a need to develop a 
better understanding of the skills reguirements for the 
development, operation and use of HTA models. The 
gualitative interviews highlighted a number of barriers 
to model checking. However, it was indicated that 
increasing time and resources would not necessarily 
improve model checking activities without a matched 
increase in their prioritisation. The authors take the 
view, as supported within the methods literature, that 
it is the interaction between the modeller and client in 
developing mutual understanding of a model that 
establishes a model's significance and its warranty. 
This position highlights that model credibility is the 
central concern to decision makers 
in using models. It is crucial then that 
the concept of model validation should not be 
externalised from the decision-makers and the 
decision-making process. 
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as paramount. 
However, good 
evidence-based care is 
more than choices. 

We discuss six 
potential 'biases' in 
EBM that may 
inadvertently devalue 
the patient and carer 
agenda To reduce 
these 'biases', EBM 
should embrace 
patient involvement in 
research, make more 
systematic use of 
individual ('personally 
significant') evidence, 
take a more 
interdisciplinary and 
humanistic view of 
consultations, address 
unequal power 
dynamics in 
healthcare encounters, 
support patient 
communities, and 
address the inverse 
care law. 

•a focus on people who seek and obtain care (rather 
than the hidden denominator of those that do not 
seek or cannot access care). 
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Additional information 


An interview with NICE 

The following passage is an edited version of an interview, conducted with a NICE employee 
about public involvement in the work of NICE. It has been shared with their permission, but 
anonymised. 

Significant sentences are in bold. 

What do you feel has been the most effective way of Involving the public? 
What has been most challenging? 

The most effective way has been involving patients, people who use services, carers and other 
members of the public as equal partners in the development of our NICE guidance. This means 
that they are appointed to NICE'S independent committees alongside professional members 
to help NICE to develop our guidance - they have equal status to professional committee 
members and play a full role in interrogating the evidence so that they can use their unique 
viewpoint to inform the deliberations of the committee that result in NICE'S recommendations. 

I would say that this has also been the most challenging strand of our public involvement 
methods - the principle of equal status for lay members on a NICE committee is something 
that we really need to keep a close eye on - chatting regularly to the lay members to make 
sure that they feel confident and able to contribute, as well as feeling that their contributions 
are valued by NICE and the professional committee members. The difference that a good chair 
makes to a lay member's confidence and contributions is often quite startling - so it's 
important for us to develop good relationships with the teams at NICE who work with their 
committee, and their chairs so that we can address any issues as and when they arise. 

We would also dearly love to have more lay members on each NICE committee, as this also 
plays a big part in increasing confidence and contributions from a lay perspective, but 
unfortunately we usually have 2 - 3 lay members on a committee of 1 5 people, which means 
that lay members are often quite heavily outnumbered by professional members. 

How do you support the public (patients, carers etc) to get involved? (e.g. 

Do you have any specific training, financial support or is it more one on one 
support provided by your colleagues?) 

Yes to all three! We aim to provide training for lay members wherever possible, and the content 
of these varies depending on the NICE work programme that the lay member has been 
appointed to work with. As an example, with NICE guidelines covering clinical and social care, 
our training workshop for new lay members covers the principles and methodology of 
guideline development, the basics of health economics, a previous example of how 
recommendations have been developed, as well as the opportunity to hear from lay members 


who have already been through this process and can therefore share really helpful hints and 
tips with the new lay members. 


In terms of financial support, we offer an attendance fee to lay members (£1 50 for a full day 
meeting, or £75 for a half day meeting), as well as covering travel and accommodation, carer 
and subsistence expenses. The attendance fee is only offered to lay members on a NICE 
committee - in recognition of the fact that their contribution is valuable and because 
professional members very often work with NICE as part of their day job and professional 
development. 


The Public Involvement Programme at NICE (PIP - which is the team that I work for) supports 
lay members, as well as the organisations who advocate for their interests, to get involved in 
the development of NICE guidance. Part of this means that someone in the PIP is assigned to 
be the public involvement lead for each and every lay member who works with NICE. This 
means that we speak with them at key points in the guidance development process, as well as 
always being available for them to get in touch with to discuss any concerns that they think 
we may be able to help with. 


It's basically about relating the principles of guidance development in a way that makes sense 
to someone who doesn't usually work with evidence as part of their everyday life. As part of 
this support, we also usually have something called a 'Welcome Pack' which we send to newly 
appointed lay members - it covers tips for success, a glossary of terms that they are likely to 
come across in evidence reviews, a list of useful websites that they can use to undertake 
further research before each meeting, as well as an FAQ document about committee 
meetings (covering things about what to expect in a committee meeting and a how to guide). 


How do you think public involvement in the whole process could be 
improved? Is there anywhere the public shouldn't or can't be involved? 

No public involvement methodology is ever perfect. It's about constantly reflecting on the 
experiences and outcomes of involvement for members of the public, and using that feedback 
to adjust our processes so that they meet the needs of lay members and their organisations 
as best they can. I've already talked about this above, but improvement for me lies in proper 
training for NICE committee chairs, making sure that our support processes are carefully 
tailored to the individual lay member depending on their own unigue needs, ensuring that NICE 
committees have a respect and appreciation for the contribution of their lay member 
colleagues, and making sure that the NICE centres responsible for selecting lay members to 
join their committees truly know and understand what makes a good lay member so that we 
are working with people who are truly passionate about what they are doing and are able to 
make a significant contribution to the work of the committee. 
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I don't believe that there is any part of NICE'S processes where patients, people who use 
services, carers and other members of the public should not be involved. However, in an 
organisation that is locked in by tight time constraints, true and authentic co-production still 
seems some distance away. For example, I think that it would be incredibly worthwhile to 
involve lay members in all of NICE'S scoping phases - which would mean that they are 
involved in determining what the guidance will and, importantly, won't cover. For clinical and 
public health guidelines, this phase usually happens before committee members are 
appointed for a particular topic. We therefore currently rely on organisations who advocate for 
the interests of the public to take part in consultations on the draft scope - which is effective 
to an extent, although I anticipate will become more problematic as voluntary and community 
sector organisations move towards an ever tightening squeeze on their resources, which is 
clearly going to impact on their ability to take part in the NICE guidance development process. 

For the moment then, it's about working with NICE'S lay members to demonstrate exactly 
what it is that the lay perspective adds to guidance development and just how effective it can 
be when it's done well. In this way, we will be able to shape and refine NICE'S processes and 
methodologies as we move forward for the benefit of the people most affected by our 
guidance. 

There are also particular difficulties with involving vulnerable groups in NICE'S work - for 
example, we don't seem to do particularly well in attracting lay member applications from 
children and young people, and expecting people with profound learning difficulties to take 
part in NICE committees would be a stretch too far for them. However there are ways around 
these kinds of things - in a situation where it had not been possible to recruit a lay member 
with personal experience of the issue or condition under consideration, we would advise the 
NICE teams to undertake some focused group work with these people to get their insight 
either in to our ideas for draft recommendations and what they would mean for them, or to 
ask for their opinions on what matters most to them in a particular treatment or care pathway 
where there is a lack of published evidence available for the committee to glean this 
information from. 

If you were setting up public involvement in the HTA process, what 
would you do? 

Other than NICE, I believe that CADTH in Canada and the SMC in Scotland do public 
involvement in the HTA process well. If I were to have the indulgence of setting up my own 
HTA process, I'd come up with a hybrid of the three as a starting point. 


• My preferred pointers for a possible hybrid of the three would be: 

• All three have a dedicated patient support team. 

• Patient participation in scoping (NICE) 

• Guide and one on one help with patient group submissions - (SMC) 

• Lay members presenting the patient evidence at committee happens at all three, 
although SMC and CADTH highlight this more than NICE - they both have specific 
time or a slot set aside for the patient views and preferences. 

• 'Patient & Clinician Engagement' meetings for ultra-orphan and end-of-life treatments 
(SMC) 

• Patient experts at the committee meeting - and one on one support for them - (NICE) 

• Consultation (NICE) 

• Right to Appeal (NICE) 
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• Feedback letter to patient groups about their submissions - this is really good 
(CADTH) 

Health Technology Assessment international 

Health Technology Assessment international have an online 'Citizen and Patient Involvement 
library (CPIL)' 119 . It is a useful collection of literature, divided into subject headings such as 
'Citizen deliberations, Patient and public involvement' and 'Patient and public involvement', 
however, the section titled 'Framework, models, roles' returns no results. 


119 http://web.archive.org/web/201 51 1 24001 71 6/http://vortal.htai.org/?g=cpil 


144 Consumers Health Forum of Australia 


Accessing documents 


This document and all those associated with this report can be accessed using the following 
link: 

https://aoo.gl/QRPVpS 
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